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Introduction 

The  Artificial  Intelligence  in  Management  (AIM)  Laboratory  of 
the  Katz  School  of  Business,  University  of  Pittsburgh  will 
employ  AIM  Lab  personnel  and  consultants  to  design,  develop  and 
test  a  prototype  of  the  following  US  Army  Medical  Research  and 
Materiel  Command  (USAMRMC) ,  Telemedicine  and  Advanced  Technology 
Research  Center  ( TATRC ) ,  Knowledge  Engineering  Group  (KEG) 
Artificial  Intelligence  Research  and  Development  efforts: 

Task  1:  Design,  Development  and  Prototyping  The  Expert 
Consult  Broker  of  GGTS 

The  AIM  Lab  will  employ  a  combination  of  cognitive  science, 
computer  science.,  artificial  intelligence,  and  management 
science  methodologies  to  perform  the  task.  The  Principal 
Investigator  will  ensure  that  each  task  is  analyzed  and  broken 
down  into  individual  components  which  will  be  then  further 
analyzed  for  specificity  and  applicability.  Once  this  is 
accomplished,  the  principal  investigator  will  apply  current  and 
state-of-the-art  processes  to  improve  each  component.  Once 
accomplished,  the  task  will  be  reengineered  in  an  iterative 
fashion . 


Body 


4 


As  detailed  in  section  5.1  of  the  approved  Statement  of  Work, 

Phase  I  of  award  DAMD  17-99-1-9581  included  six  tasks: 

5.1.1  Develop  a  conceptual  model  of  a  telemedicine  consult 
management  system  operating  within  a  dynamic  military 
environment.  The  model  will  address  the  ways  in  which 
consult  routing  could  occur,  as  in  CB-LRCE.  Wherever 
possible,  the  consult  broker  will  access  information 
regarding  dynamic  environmental  aspects  by  querying  the 
Combat  Service  Support  Control  System  -  Medical  component 
of  the  Army  Global  Command  and  Control  System.  The  model 
should  be  reflective  of  current  worldwide  medical 
consultation  management  practices  and  past  experience  in 
both  the  military  and  civilian  sectors. 

5.1.2  Select  one  or  more  knowledge  representation  schemes  to 
implement  the  model. 

5.1.3  Instantiate  sufficient  domain  knowledge  to  enable  a 

structured  consultation  process  walk-through  to  be 

demonstrated  to  a  peer  review  panel  composed  of  physicians 
and  knowledge  engineering  professionals  from  military  and 
civilian  academic  organizations. 

5.1.4  Demonstrate  that  the  conceptual  model  of  5.1.1  together 
with  the  domain  knowledge  of  5.1.3  represented  as  in  5.1.2 
is  capable  of  functioning  as  an  effective  and  reliable 
consult  manager. 

5.1.5  Prepare  and  deliver  expert  system  model  specification. 
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5.1.6  Prepare  and  deliver  verification,  validation  and  operation 
prototype  test  plans. 

In  this  part  of  our  report,  we  describe  our  accomplishments 
associated  with  each  of  those  tasks.  Six  documents,  attached  as 
appendices,  provide  the  details  referred  to  in  this  section. 

Those  documents  are : 

Appendix  1.  Gilbert,  May,  Vargas,  and  Brienza,  "The  Global 
Grid  Telemedicine  System  Consult  Broker, "  AIM  Lab  Working 
Paper  number  101; 

Appendix  2.  May,  Vargas,  Johnson,  Gilbert,  and  Illi,  "Global 
Grid  Telemedicine  System, ”  presented  at  ATA  Fourth  Annual 
Meeting,  Salt  Lake  City,  Utah,  1999; 

Appendix  3.  May,  Vargas,  Gilbert,  Illi,  Rocca,  and  Jacobs, 

" GGTS  Consult  Broker, "  presented  at  the  ATA  Fifth  Annual 
Meeting,  Phoenix,  Arizona,  2000; 

Appendix  4.  Gilbert,  Illi,  Baker,  Brienza,  Rocca;  Vargas,  and 
May,  "Global  Grid  Telemedicine  System  (GGTS) , "  presented  at 
the  ATA  Fifth  Annual  Meeting,  Phoenix,  Arizona,  2000; 

Appendix  5.  Summary  of  naive  text  classification  experiments 
on  teledental  and  medical  data;  and 

Appendix  6.  Code  listing  for  Teledental  Consult  Broker 
Prototype . 

We  chose  to  pursue  a  data  driven  approach  to  Tasks  5.1.1  through 
5.1.6  rather  than  a  top-down  system  development  methodology.  In 
other  words,  we  decided  to  use  whatever  information  could  be 
found  that  related  to  at  least  partly  fielded  and  used  Army 
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telemedical  systems  as  the  basis  for  developing  a  prototype  in  a 
narrow  application  area.  Such  a  narrow-but-deep  strategy  could 
help  us  appreciate  and  integrate  characteristics  that  are 
critical  for  system  acceptance  and  functionality.  The  primary 
risk  of  our  strategy  is  that  what  we  learn  from  a  narrow 
application  domain  may  not  generalize  to  other  parts  of 
telemedicine,  but  we  considered  the  benefits  of  being  able  to 
rapidly  develop  a  prototype  and  to  learn  from  its  construction 
made  that  an  acceptable  risk. 

In  order  to  pursue  a  data-driven  strategy,  we  required  data. 
The  group  at  TATRC  that  was  associated  with  the  Army's 
Teledental  system  had  about  500  e-mail  records  from  their 
Internet-based  system,  as  was  willing  to  share  those  records, 
properly  sanitized,  with  us.  We  began  to  receive  parts  of  their 
data  at  the  beginning  of  January  2  000,  and  had  a  complete  data 
set  approximately  one  month  later.  The  teledental  data  included 
the  text  from  Form  513  for  each  patient  and  the  location  and 
code  number  for  the  consultant  who  responded  to  the  request,  but 
not  the  reason (s)  why  that  particular  consultant  was  chosen  or 
chose  to  respond. 

Task  5.1.1. 

A  preliminary  conceptual  model  for  the  Consult  Broker,  including 
four  possible  interface  designs,  is  described  in  Appendix  2. 
After  receipt  of  the  Teledental  data,  we  further  developed  that 
model,  specialized  in  part  to  characteristics  of  the  Teledental 
environment.  The  refined  model  is  discussed  in  Appendix  3, 
Appendix  4,  and  Appendix  1.  The  computer  expression  of  the 
conceptual  model  is  fully  described  in  Appendix  6.  The 
interrelationships  between  the  Consult  Broker,  and  a  possible 
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network  environment  are  discussed  in  Appendix  5  and  in  Booz- 
Allen  Hamilton  (2000)  . 1 

Task  5.1.2. 

We  chose  to  represent  knowledge  structures  such  as  consultants, 
units,  and  facilities  using  objects,  and  data  as  instances  of 
those  objects.  A  text  description  of  the  representation  is 
included  in  Appendix  1,  and  its  computer  realization  in  Appendix 
6 . 

Task  5.1.3 

Based  on  analyses  of  the  Teledental  data  and  discussions  with 
TATRC  personnel,  we  first  identified  three  distinct  types  of 
teledental  consultations. 

Type  1:  The  requester  is  primarily  interested  in  referring 
the  patient  to  the  consultant,  and  has  asked  for  a  consultation 
in  order  to  find  the  best  match  for  that  referral.  The  primary 
goal  of  the  requester  is  thus  administrative,  not  medical. 

Type  2 :  The  requester  fully  intends  to  treat  the  patient 
himself,  but  has  encountered  complications  or  problems  that  are 
outside  of  his  experience  or  expertise.  The  purpose  of  the 
consultation  is  to  give  him  the  additional  information  necessary 
to  permit  him  to  carry  out  his  treatment  of  the  patient.  The 
primary  goal  of  the  consultation  is  medical,  not  administrative. 

Type  3 :  The  type  of  request  is  intermediate  between  types 
1  and  2 .  The  request  is  initially  medically  oriented,  but  the 
requestor  will  continue  treatment  only  if  the  problem  turns  out 
to  be  within  his  expertise,  even  after  having  been  provided  with 
additional  information  from  the  consultant.  If  the  problem  is 
too  difficult  for  the  requester,  he  is  interested  in  referring 
the  patient  to  the  consultant  for  further  treatment. 


1  Booz-Allen  Hamilton  (2000),  “GGTS  Implementation  Plan.” 


8 


The  type  of  consultation  affects  both  the  way  in  which  the 
Consult  Broker  should  be  expected  to  interact  with  the  requester 
and  the  amount  and  type  of  information  the  requester  can  be 
expected  to  provide  to  the  Consult  Broker.  In  a  Type  1 
consultation,  geographic  proximity  of  the  consultant  to  the 
requester  could  be  very  important,  because  the  patient  is  going 
to  be  handed  off  from  the  requester  to  the  consultant.  Depth  of 
expertise  on  the  part  of  the  consultant  is  much  more  salient  in 
a  Type  2  consultation.  A  request  for  a  Type  2  consultation  is 
more  likely  to  have  specific  likely  procedures  identified  than 
would  be  a  Type  1  consult.  For  example,  a  Type  1  teledental 
consult  might  be  as  simple  as  a  request  to  find  the  nearest  oral 
surgeon  so  that  a  patient  can  have  an  impacted  wisdom  tooth 
extracted.  In  a  Type  2  consult,  the  requester  is  more  likely  to 
be  able  to  provide  the  specific  CDT  procedure  code(s)  the 
patient  would  require. 

As  detailed  in  Appendix  3,  we  developed  six  scenarios  for 
possible  teledental  consultations.  They  are: 

Scenario  1:  Peacetime,  with  consultants  in  their  default 
locations,  and  the  requester  able  to  provide  as  least  the 
specialty  required 

Scenario  2:  Conflict,  in  which  deployable  units  may  be 
located  in  other  than  their  default  locations,  but  with 
peacetime  organizational  rules 

Scenario  3:  Scenario  2,  but  with  special  organizational 
rule  for  communication  and  patient  movements 

Scenario  4:  A  Type  2  consult  in  which  there  is  a  trade 
off  between  immediate  consultant  availability  and  specific  prior 
experience 

Scenario  5:  A  consult  in  which  the  specific  expertise 
requested  is  new  to  the  system,  and  the  Consult  Broker  has  to 
use  its  knowledge  of  dental  procedures  to  identify  patients  most 
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similar  to  the  new  consultation  (case-based  reasoning  on  patient 
records ) 

Scenario  6:  A  consult  request  in  which  the  requestor  is 

not  exactly  certain  of  the  specific  expertise  necessary,  so  that 
the  Consult  Broker  is  expected  to  provide  the  names  of 
consultants  with  similar  expertise  as  well  as  those  with  exactly 
the  expertise  requested  (case-based  reasoning  on  consultants)  . 

Appendix  3  shows  the  prototype  system's  behavior,  and  the 
interface  screens  that  result,  from  all  six  scenarios.  The 
consultation  process  for  those  scenarios  was  demonstrated  to  an 
audience  of  physicians  and  knowledge  engineering  professionals 
at  the  ATA  2000  national  meeting  in  Phoenix,  AZ . 

Task  5.1.4. 

The  demonstration  of  effectiveness  and  reliability  of  the 
conceptual  model  and  its  computer  implementation  as  a  stand¬ 
alone  system  would  require  our  having  access  to  a  data  set  in 
which  the  choice  of  consultants  to  match  requests  was  based  on 
the  criteria  desired  by  the  Army,  and  for  which  those  criteria 
were  fully  disclosed  to  us.  To  demonstrate  its  effectiveness  in 
a  larger  network  environment  would  require  the  development  of  at 
least  a  prototype  of  that  larger  environment. 

Task  5.1.5 

The  model  specification,  for  at  least  the  teledental 
application,  is  described  in  Appendix  1  and  Appendix  6. 

Task  5.1.6 


This  task  also  requires  that  we  be  provided  with  suitable  data. 


Key  Research  Accomplishments 


•  Development  of  a  prototype  teledental  consult  broker 

•  A  metric  for  case-based  reasoning  based  on  a 
multidimensional  scaling  of  similarity  judgments  of 
procedures 

•  A  naive  Bayesian  classifier  for  extraction  of 
administrative  information  from  technical  medical  text 
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Reportable  Outcomes 

Gilbert,  May,  Vargas,  and  Brienza,  "The  Global  Grid  Telemedicine 
System  Consult  Broker, "  AIM  Lab  Working  Paper  number 

May,  Vargas,  Johnson,  Gilbert,  and  Illi,  "Global  Grid 
Telemedicine  System, "  presented  at  ATA  Fourth  Annual  Meeting, 
Salt  Lake  City,  Utah,  1999, 

May,  Vargas,  Gilbert,  Illi,  Rocca,  and  Jacobs,  " GGTS  Consult 
Broker, "  presented  at  the  ATA  Fifth  Annual  Meeting,  Phoenix, 
Arizona,  2000, 

Gilbert,  Illi,  Baker,  Brienza,  Rocca;  Vargas,  and  May,  "Global 
Grid  Telemedicine  System  (GGTS) , "  presented  at  the  ATA  Fifth 
Annual  Meeting,  Phoenix,  Arizona,  2000, 

Summary  of  naive  text  classification  experiments  on  teledental 
and  medical  data,  and 

Code  listing  for  Teledental  Consult  Broker  Prototype. 
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Conclusions 

Our  progress  to  date  on  this  award  demonstrates  that  we 
have  been  able  to  conceptualize,  formalize,  and  implement  the 
technical  aspects  of  a  Consult  Broker  for  the  Global  Grid 
Telemedicine  System.  Using  techniques  from  artificial 

intelligence  and  statistics,  we  have  successfully  represented 
both  data  and  knowledge  for  a  teledental  prototype,  and  have 
shown  that  our  prototype  is  capable  of  supporting  decision 
making  in  a  variety  of  scenarios.  We  have  also  begun  the 
process  of  generalizing  the  teledental  model  to  the  more  general 
medical  domain,  primarily  through  our  experiments  in  using  naive 
classification  to  extract  what  an  administrative  system,  such  as 
the  Consult  Broker,  needs  to  know  from  technical  medical  text. 

Progress  beyond  this  point  in  the  development  of  the 
Consult  Broker  will  be  limited  by  the  availability  of  adequate 
data  and  adequate  administrative  knowledge,  rather  than  by 
technical  difficulties  associated  with  implementation.  With  the 
apparent  hiatus  in  further  work  on  the  originally  envisioned 
network  environment  for  the  GGTS ,  a  specific  delineation  of  the 
place  and  role  of  the  Consult  Broker  within  the  larger  Global 
Grid  environment  is  necessary.  Such  a  delineation  would  define 
what  type  of  telemedical  systems  would  be  supported  by  the 
Consult  Broker,  and  the  way  in  which  users  would  be  expected  to 
interact  with  the  Consult  Broker.  If  those  systems  are  ones 
that  are  at  least  partly  implemented,  then  data  from  them  and  a 
thorough  description  of  the  medical  administration  of  the 
teleconsultation  processes  associated  with  them  is  critical  for 
Consult  Broker  development,  testing,  verification,  and 
validation.  If  systems  to  be  supported  by  the  Consult  Broker 
are  also  only  at  the  planning  or  prototype  stage,  then 
descriptions  of  those  systems '  intended  architecture  and  the 
test  cases  to  be  used  for  their  testing,  verification,  and 
validation,  would  aid  permit  us  to  design  a  Consult  Broker  that 
would  be  compatible  with  those  systems. 


Appendix  1: 


THE  GLOBAL  GRID  TELEMEDICINE  SYSTEM  CONSULT  BROKER* 


Gary  R.  Gilbert 

University  of  Pittsburgh  and  USAMR&MC\TATRC 

Jerrold  H .  May 
Luis  G.  Vargas 

Katz  Graduate  School  of  Business 
AIM  Laboratory 
University  of  Pittsburgh 


*  This  research  has  been  supported  by  grant  number  DAMD  17-99-1-9581  to  the 
University  of  Pittsburgh. 
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1.  INTRODUCTION 


The  Global  Grid  Telemedicine  System  (GGTS)  is  a 
proposed  global  telemedicine  command  and  control  system 
(Zajtchuk  and  Zajtchuk,  1996)  that  will  enable  telemedical 
consultations  to  occur  anywhere  in  the  world,  regardless  of 
location  or  transportation  medium.  The  objectives  are  to: 
(1)  leverage  the  extensive  military  national  command 
communications  facilities  throughout  the  world  to  provide 
global  telemedicine  support  to  US  and  allied  forces, 
worldwide;  (2)  provide  a  transparent  fully-integrated 
commercial  communications  network  backup  to  military 
facilities;  and  (3)  develop  a  computer  assisted  consultation 
routing  system  that  maximizes  world-wide  projection  of 
consultation  support  from  medical  consultants  and  sub¬ 
specialists  located  at  military  and  civilian  medical  centers 
throughout  the  United  States.  The  concept  envisions 
sophisticated  network  management  tools  that  maximize 
efficient  use  of  military  and  civilian  communications 
infrastructure  at  bandwidths,  protocols,  security  levels, 
and  modalities  appropriate  to  specific  telemedicine 
applications.  A  key  component  is  an  intelligent  global 
teleconsultation  routing  and  management  system  capable  of 
determining,  from  very  minimal  requester  provided 
information,  the  type  and  location  of  consultant  needed,  the 
protocol  conversions  required,  the  priority  of  need,  and  the 
optimal  network  routing  to  facilitate  connectivity. 

Concept  for  Global  Grid  Telemedicine  System 

For  the  last  several  years,  the  Department  of  Defense 
has  implemented  numerous  rapid  prototyping  and  integration 
efforts  as  part  of  the  DoD  Telemedicine  Testbed  at  Fort 
Detrick,  Maryland.  These  prototypes  were  continuously 
evaluated  in  Joint  Technology  Demonstrations,  in  Army 
Battle  Labs  Advanced  Warfighting  Experiments  and  other 
joint  service  exercises.  Contemporary  military  operations 
in  the  Balkans  and  the  Middle  East  have  also  offered  real- 
world  environments  to  validate  emerging  telemedicine 
technologies.  Although  much  work  has  been  done,  the 
military  health  system  currently  does  not  have  an  effective 
method  of  linking  health  care  providers  and  diagnostic 
systems  into  a  seamless  diagnostic,  treatment,  and 
evacuation  capability.  Health  care  personnel  at  all 
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echelons  of  care  must  be  able  to  communicate  with  each 
other  by  audio,  video,  and  electronic  media,  so  maximum 
utilization  of  expert  consultant  skills,  diagnostic 
capability,  and  treatment  regimens  can  be  quickly 
implemented  to  provide  state-of-the-art  resuscitation, 
care,  and  enhanced  evacuation.  Integration  of  existing  and 
emerging  digital  communication  technologies  into  the 
patient  care  system,  beginning  with  the  individual  soldier 
and  continuing  throughout  the  health  care  continuum,  will 
allow  the  military  health  system  to  project  expert 
preventive  medicine  and  treatment  necessary  to  improve  the 
sustainability  of  the  future  military  force.  In  addition, 
health  care  in  special  operations  and  military  operations 
other  than  war  (e.g.  humanitarian  and  peacekeeping 
missions)  will  be  enhanced  by  the  utilization  of  these 
technological  innovations.  The  timing  of  this  need  is 
based  on  ongoing  joint  telemedicine  advanced  technology 
demonstrations  and  advanced  warfighting  experiments  that 
have  demonstrated  that  integrating  emerging  information 
management  technologies  into  the  health  services  mission 
area  will  significantly  improve  combat  casualty  care. 


Traditionally,  the  military  medical  community  has  had  very 
limited  access  to  tactical  and  strategic  communications 
assets.  In  addition,  the  increased  employment  of 
telemedicine  technologies  by  the  US.  military  has  led  to 
haphazard  growth  of  stovepiped,  leased  commercial 
communications  networks  to  support  military  telemedicine 
applications.  These  networks  are  excessively  expensive, 
inefficiently  designed  to  meet  global-needs,  and  ill- 
equipped  to  assist  end-users  in  routing  teleconsultation 
and  medical  messaging  traffic  to  appropriate  destinations. 

As  global  telemedicine  operations  are  incorporated  into  US 
military  medical  doctrine  for  world-wide  deployments, 
increased  access  to  tactical  and  strategic  communications 
assets  for  the  military  medical  community  is  becoming 
essential . 

In  order  to  address  these  changing  operational 
requirements  and  to  enable  the  DoD  to  leverage  rapidly 
emerging  technological  innovations,  the  concept  of  a  Global 
Grid  Telemedicine  System  (GGTS)  was  conceived.  The  GGTS 
will  match,  in  real  time  or  store-and-forward  mode,  remote 
or  deployed  medical  personnel  or  facilities  requesting 
telemedicine  consultation  support  with  provider  facilities 
at  major  military  and  civilian  medical  centers.  It  also 
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establishes  and  manages  the  connectivity  required  to 
support  the  consultation  session  as  well  as  security  fire 
walls  to  enable  communications  from  forward  tactical  forces 
including  special  operations  forces  operating  well  behind 
enemy  lines.  Medical  sub-specialty  consultants  are 
generally  based  at  medical  centers  in  the  United  States  and 
accessible  through  a  variety  of  commercial  or  defense 
networks  if  adequate  multilevel  security  features  can 
enable  connectivity .  The  two  key  components  of  the 
proposed  GGTS  system  are  the  Consultation  Broker  Server, 
and  the  managed  Communications  Networks  consisting  of  a 
Requestor  Network  on  one  side  of  the  Consultation  Broker, 
and  a  Provider  Network  on  the  other  side.  The  primary 
function  of  the  GGTS  is  to  match  remote,  deployed  medical 
facilities  requesting  telemedicine  consultation  support 
with  provider  facilities  around  the  world,  and  to  establish 
and  manage  the  connectivity  required  to  support  the 
consultation  session.  Consult  providers,  individual  sub¬ 
specialty  physicians  at  primary  medical  facilities, 
"register"  with  the  GGTS  Consult  Broker  when  they  are 
available  to  receive  consult  requests  within  their  area  of 
expertise.  Remote  forward  deployed  medical  facilities 
(afloat  or  land-based)  submit  consult  requests  to  the  GGTS 
Consult  Broker  specifying  the  classification  of  the  medical 
issue  and  the  type  of  telemedicine  session  required.  The 
Consult  Broker  will  then  match  the  request  with  the 
appropriate  registered  provider  and  establish  or  schedule  a 
real-time  consultation  session  between  the  two  nodes 
(Provider  Node  and  Requestor  Node)  or  forward  a  store-and- 
forward  consultation  request  to  an  appropriate  consultant 
and  return  the  results  to  the  requestor.  It  will  also 
manage  the  Requestor  and  Provider  Network  communications 
resources  available  to  the  GGTS  to  provide  optimal 
bandwidth,  quality  of  service,  and  security  for  concurrent 
telemedicine  sessions.  Because  of  rapidly  changing 
bandwidth  requirements  for  strategic  communications 
throughout  the  entire  DoD,  the  GGTS  network  will  be 
designed  to  leverage  military  bandwidth  when  available  and 
transparently  switch  to  commercial  bandwidth  on-demand  when 
other  DoD  strategic  and/or  tactical  priorities  preempt 
telemedicine  applications. 


This  paper  concentrates  on  the  Consultation  Broker  (CB) 
component  of  the  GGTS  operationalized  using  Artificial 
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Intelligence  methodologies.  The  CB  is  an  object  oriented 
global  teleconsultation  system  designed  to  route  incoming 
requests  for  consultation  to  the  appropriate  "on  duty" 
medical  personnel.  The  CB  must  have  sufficient  ingrained 
"intelligence"  to  determine  -  in  real  time-  the  type  of 
consultant  needed  from  very  minimal  requester  provided 
information,  and  to  determine  protocol  conversions  needed, 
priority  of  need,  and  optimal  network  routing  to  facilitate 
connectivity. 

Previous  research  within  the  Army  indicates  that 
telemedical  practice  requires  "standards  of  utilization" 
(Walters,  1996) .  Significant  controversy  exists  over  what 
value  various  telemedical  modalities  offer  to  the 
consultation  process,  what  minimum  information  is  needed  for 
consultations,  and  how  that  information  should  be  presented 
to  the  consultant.  There  are  also  debates  over  which 
medical  sub-specialties  require  what  information  media  or 
telemedical  modalities:  store-and-forward  versus  real-time 
video  consultations,  high  resolution  still  images  versus 
full  motion  video  clips,  radiology  and  pathology  images 
versus  radiology  and  pathology  reports,  and  telephone 
consultations  versus  e-mail.  Calcagni  et  al .  (1996)  and 
Clyburn  et  al.  (1998)  provide  some  experiences  with  the  use 
of  telemedicine.  However,  insufficient  empirical  data  or 
hard  analysis  yet  exists  to  make  definitive  recommendations 
by  sub-specialty.  Therefore  an  important  function  of  the  CB 
system  should  be  to  collect  and  analyze  information  on  the 
telemedicine  consultation  process. 

Walters  (1996)  identified  a  number  of  objective  and 
subjective  questions  that  should  be  asked  about  each 
telemedicine  consultation  in  order  to  determine  how  best  to 
execute  it.  In  addition  to  considering  what  telemedical 
modalities  are  appropriate  for  each  case,  Walters'  list  of 
questions  includes  basic  patient  demographic  data  (e.g., 
name,  date,  location,  age,  and  nationality) ,  and 
consultation  specific  data  (e.g.,  the  requesting 
physician's  specialty,  what  he/she  is  asking  the 
consultant,  consultation  urgency,  subspecialties  requested, 
type  of  equipment  used  or  available,  history  of  present 
illness,  and  current  differential  diagnosis) .  Walters 
also  extracted  and  retained  information  from  the 
consultant's  reply,  such  as  the  adjusted  diagnosis  and 
treatment  recommendations.  In  her  research,  Walters 
employed  a  physician  reviewer  who  addressed  questions  such 
as,  How  sick  was  the  patient?  Did  the  consultation  change 
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the  diagnosis,  treatment,  or  duty  status  of  the  patient  or 
effect  the  outcome?  What  type  of  communication  modality 
was  needed  by  the  consultant  to  reply  to  the  consultation? 
Walters  found  that  two  patterns  of  consultation  referral 
developed.  The  first  involved  physicians  consulting  on 
conditions  outside  of  their  own  specialty.  The  second 
involved  specialty  physicians  consulting  with 
subspecialists  or  specialists  with  greater  expertise  in  the 
requestor's  own  specialties. 

Reporting  on  the  Army  telemedicine  experience  in  the 
republics  of  the  former  Yugoslavia,  Navein  et  al.  (1996) 
observed  that  quality  of  service  (response  time,  quality  of 
consultation,  expertise  of  consultants,  etc.)  was  the  most 
significant  factor  in  the  choice  of  consultant  when 
consultation  routing  was  left  to  the  prerogatives  of 
requesting  physicians  alone.  Even  though  the  Army 
considered  that  "best  practice"  would  dictate  that 
consultations  be  directed  to  the  "referral  center"  hospital 
to  which  it  wanted  patients  to  be  evacuated,  deployed 
physicians  generally  directed  consults  to  the  medical 
center  providing  the  best  telemedicine  service. 

2.  THE  NETWORK  ENVIRONMENT 

Booze-Alien  Hamilton  (1999?)  observed  that  the  GGTS, 
at  its  highest  level,  is  a  system  that  manages  a  medical 
information  exchange  network  in  the  support  key  military 
health  service  support  processes.  The  GGTS  accomplishes 
this  through  the  use  of  4  distinct  elements  operating 
through  a  managed  interface  to  the  network.  In  addition  an 
underlying  GGTS  communications  network  is  employed  to 
facilitate  the  interaction  between  each  of  the  elements  of 
GGTS.  Figure  1  shows  the  relationship  between  five  health 
service  support  "pillars",  the  medical  support  net,  and  the 
GGTS  network,  elements. 

The  GGTS  is  composed  of  four  distinct  functional 
nodes:  Field,  Requestor,  Broker  and  Provider.  Each  one  of 
these  nodes  provides  a  specific  interface  to  the 
communications  network  that  supports  medical  operations 
across  the  Operational  Continuum.  In  addition  to  providing 
interfaces,  each  node  also  contains  a  specific  set  of 
functionality  that  helps  enable  the  GGTS  to  manage  the 
communications  network. 
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Figure  1.  GGTS  Support  to  Five  Medical  Business 
Processes  and  Interfaces. 

The  Field  Node  is  most  commonly  associated  with  the 
end-user  of  GGTS.  The  end-user  can  anyone  from  a  first 
responder  and  forward  deployed  medics  to  medical  command 
and  control  and  medical  space  awareness  personnel.  It  is 
the  Field  Node  that  provides  the  initial  gateway  through 
which  medical  information  and  services  may  be  requested. 
This  node  accomplishes  three  tasks: 

•  Gathering  medically  relevant  information  from  the 
patients  and  facilities. 

•  Make  available  relevant  information  about  patients  and 
facilities  ,  and 

•  Initiate  Tele-medicine  Consultation  Sessions  by 
establishing  a  connection  with  the  GGTS  Requester 
Node . 

The  Requestor  Node's  primary  duty  is  to  act  as  the 
central  gathering  point  for  Field  Node  requests  and  to 
maintain  communications  links  with  the  Broker  and  each 
Field  Node  it  is  responsible  for.  Thus  requestor  acts  a 
form  of  mediator  between  the  Broker  and  each  Field  Node  for 
which  it  is  responsible  This  node  serves  as  the  Central 
point  of  contact  for  all  Field  Nodes  in  the  assigned  area 
of  responsibility  by  processing  User  Requests  to  Start 
Applications,  retrieve  information  and  begin  Tele¬ 
consultations,  maintain  communication  with  Broker  Node  and 
Field  Node/Units,  reallocate  bandwidth  among  numerous  Field 
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Nodes,  maintain  the  Status  of  the  Communications  Equipment, 
Format/Transmit  Request  Messages  to  Consult  Manager  in 
Broker  Node,  and  if  possible  assign  Medical  Specialty  and 
establish  the  required  and  Requested  Bandwidth  and 
Duration  of  the  consultation. 

The  Broker  Node  is  the  "Brains"  of  the  GGTS .  As  shown 
in  Figure  2  it  is  sub-divided  into  four  separate  managers. 
Each  one  being  responsible  for  a  different  management 
aspect  of  the  GGTS. 


/Network  Bandwidth^ 
Content  Manager 

V 

// - \ 

Decisions  Aides 

Module  y, 

/ 

Module  ^/) 

\  X 

//// Consult  Manager  ^ 
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/^Network  Manager^ 
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Figure  2.  GGTS  Broker  Node  Managers 

At  a  high  level,  the  Borker  Node  performs  the 
following  brief  list  of  functions : 


•  Receive  consultation  requests  from  Consultation 
Requester  Node 

•  Match  requests  with  appropriate  Provider  node 

•  Use  Medical  Artificial  Intelligence  (AI)  to  satisfy 
Field  Node  requests 

•  Network  management 

•  Determine  physical  communications  link  and  topology 

•  Establish  /  Maintain  permanent  path  between  broker  and 
requester  base  nodes. 

•  Dynamically  reallocate  bandwidth  as  required 

•  Maintain  numerous  ongoing  sessions,  and 

•  Establish  a  connection. 
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The  Network  Manager  Module  is  responsible  for 
monitoring  the  communications  networks  allocated  to  the 
GGTS.  It  provides  continuous  topology  information  to: 

•  The  Network/Bandwidth/Content  Manager  Module 

•  The  Decision  Aides  Module 

The  Network  Bandwidth/Content  Manager  Module  is  a 
multi  facetted  module  that  manages  not  only  the  bandwidth 
assigned  to  the  GGTS  but  also  the  content  of  the 
information  flowing  across  its  allocated  networks.  The 
main  functions  attributed  to  the  module  are: 

•  To  Generate/Execute  Bandwidth  Reallocation  Commands 
based  on  input  from 

the  Network  Manager  and  Consult  Manager  Modules  ; 

•  To  Monitor  and  Display  the  Status  of  the  Consultation 
Provider  Nodes, 

the  System  performance (e . g .  error  rate,  QoS)  of  On- 
Going  Consultations 

and  the  Available  Bandwidth  in  the  entire  system  ;  and 

•  To  Monitor  and  Manage  Prioritized  Content  sent  across 
the  network. 

The  Decisions  Aides  Modules  provides  the  Broker  a  set 
of  tools  and  functionalities  designed  to  help  facilitate 
its  decision  making  process.  These  tools  can  be  anything 
from  additional  rule  sets  to  automated  software 
applications  that  keep  track  of  specific  data  and 
information.  Among  some  of  the  functionalities  it  is  worth 
to  mention  we  have: 

•  Support  the  interpretation  of  relationships  between 
system  dynamics  and  user  priorities; 

•  Help  to  determine  optimal  utilization  of  available 
resources  such  as  Communication  and  Medical  Assets, 
Specific  Paths/Links,  Available  Equipment,  and 
Satellite  Footprints; 
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•  Provide  Direction  to  Network  Bandwidth  Manager  Module 
(through  Consult  Manager)  in  Allocating  Communication 
Assets; 

•  Evaluate  Consultation  Requests  Against:  Provider/Field 
Node  Profiles,  Communications  Availability,  Request 
Priorities,  and  Other  Mission  Specific  Rules. 

The  Consult  Manager  is  the  part  of  the  Broker  that 
determines  which  Medical  Providers  best  meets  the  stated 
request  from  a  Field  Node.  The  main  tasks  of  this  module 
are  to: 


•  Receive/Prioritize  Consultation  Requests; 

•  Querie  Decisions  Aides  Module  for  requested  system 
performance  (e.g.  error  rate, Quality  of  Service  (QoS), 
Availability) ; 

•  Notify  Consultation  Providers  of  incoming  Consultation 
Requests; 

•  Prepare/Transmit  Network  Configuration 
Mess ages /Responses; 

•  Coordinate  Consultation  Request  Information  among 
other  Consultation  Broker  Server  Modules;  and 


•  Determine  which  providers  to  assign  to  which 
consultation  request. 

The  Provider  Node  is  the  part  of  GGTS  that  interfaces 
with  those  locations  that  contain  the  requested  medical 
support  applications  or  medical  personnel  as  stated  in  a 
Field  Node  Requests.  It  is  the  job  of  the  Provider  Node 
to  enable  these  medical  personnel  and  applications  to 
interact  effectively  with  the  requesting  Field  Node.  It  is 
in  charge  of  : 


•  Processing  Requests  to  Start  Applications; 

•  Receiving/Processing  Incoming  Consult  Warning  Message 
and  Configuration  Messages; 


•  Configuring  Collocated  Communications  Equipment; 
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•  Hosting,  whenever  appropriate.  Telemedicine 
"Applications"  or  Interface  With  Them  as  External 
Systems; 

•  Launching  Applicable/appropriate  Telemedicine 
Applications  or  Interface  with  Them; 

•  Facilitating  medical  personal  interaction  with  the 
Consultation; 

•  Loging  all  Consultation  related  information  into  a 
database; 

•  Maintaining  Interface/Communications  with  the  Broker 
Node; 

•  Allocating  Bandwidth  among  Provider  Node  Workstations; 
and 

•  Maintaining  Provider  User  Machine  Interface. 

In  order  for  the  GGTS  to  effectively  perform  its 
functions,  each  functional  node  of  the  GGTS  needs  to  be 
able  to  communicate  with  each  other.  This  is  accomplished 
through  the  use  of  a  GGTS  network  that  enables  each  node  to 
connect  to  each  other.  Figure  3  shows  the  relationship  of 
each  of  the  functional  nodes  with  regards  to  each  other  and 
the  communications  network. 


Figure  3.  GGTS  Element  Relationship 

An  emerging  information  architecture  doctrine  for 
the  Department  of  Defense,  called  Global  Information 
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Grid  (GIG)  is  based  on  a  military  concept  called 
"Network  Centric  Warfare"  (NCW) .  NCW  is  defined  as 
"an  information  superiority-enabled  concept  of 
operations  that  generates  increased  combat  power  by 
networking  sensors,  decision  makers,  and  shooters  to 
achieve  shared  awareness".  Information  superiority  is 
the  capability  to  collect,  process,  and  disseminate  an 
uninterrupted  flow  of  information  while  exploiting  or 
denying  an  adversary's  ability  to  do  the  same.  The 
objective  of  NCW  is  to  "translate  information 
superiority  into  combat  power  by  effectively  linking 
knowledgeable  entities  in  the  battle  space".  The  GIG 
was  born  out  of  concerns  regarding  interoperability 
and  end-to-end  integration  of  automated  information 
systems  within  the  Department  of  Defense.  As  a 
unified  "system-of-systems",  the  GIG  is  the  key 
enabler  of  information  superiority.  The  GIG  is  the 
globally  interconnected,  end-to-end  set  of  information 
capabilities,  associated  processes  and  personnel  for 
collecting,  processing,  storing,  disseminating  and 
managing  information  on  demand  to  warfighters,  policy 
makers,  and  support  personnel. 

The  NCW  concept  is  also  the  basis  of  the  emerging 
"net-centric"  architecture  for  military  medical 
operations.  Net-centric  operations  military 
operations  are  built  around  command ,  control, 
communications ,  computers,  and  intelligence  (C4I) 
systems  with  the  specific  goal  of  supplementing  or 
replacing  directed  linear  point-to-point  information 
flows  with  network  centered  information  flows  similar 
to  a  message  board.  A  distributed  network  centered 
information  architecture  serves  as  both  a  collection 
center  for  information  from  those  generating  it,  as 
well  as  a  source  of  information  for  those  needing  it, 
without  a  pre-planned  or  required  communications  link 
between  information  generators  and  information  users. 
Communications  systems  are  traditionally  described  in 
terms  of  send  and  receive  architectures.  While  any 
information  flow  from  the  communications  technical 
architecture  perspective  involves  a  "send  and  receive" 
concept,  net-centric  information  flows  from  the 
operational  architecture  perspective  involve 
information  contributions  (new  inputs  and  updates)  and 
information  queries  (requests  with  accompanying 
outputs) .  Within  net-centric  operations  many  of  the 
information  deposits  will  be  of  the  "transmit  and 
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forget"  nature  (those  inputting  the  information  will 
not  be  concerned  with  who  uses  it)  ,  while  many  of  the 
information  queries  will  be  "blind"  (those  requesting 
and  using  the  information  will  not  be  aware  of  its 
source)  .  In  order  for  the  GGTS  medical  information 
and  communication  support  network  discussed  above  to 
be  efficiently  and  securely  implemented  within  the  | 
Military  Global  Information  Grid,  and  also  be  flexible 
enough  to  accommodate  spontanenous,  random,  24  hour 
consultation  requests,  and  all  four  Consultation 

Broker  implementation  alternatives  discussed  in 

Section  4  below,  a  net-centric  architecture  approach 
that  supports  "transmit  and  forget"  input  and  "blind 
query"  output,  is  essential. 

Likewise,  under  all  four  Consultation  Broker 
alternatives,  the  Consultation  Broker  must  have  a  way 
to  represent  requestor,  patient,  consultant,  and 
consultation  site  information,  with  the  level  of 
detail  necessary  dependent  in  part  on  the  alternative 
chosen,  and  a  way  to  represent  actual  and  elapsed 
time.  Much  of  the  patient  information  can  be 
described  numerically.  ICD9  diagnostic  codes  could  be 
used  by  the  requestor  to  convey  possible  patient 
problems  and  for  the  consultant  to  refine  differential 
diagnoses,  confirm  diagnoses,  or  modify  diagnoses. 
ASA  codes  could  be  used  to  describe  the  severity  of 
the  patient's  condition  and  its  rate  of  deterioration. 
Patient  demographics  (age,  sex,  etc.)  could  also  be 
communicated  either  numerically  or  by  menu  choice  that 
would  be  coded  numerically  by  the  CB.  Critical 

consultant  and  site  information  could  also  be 
represented  using  ranges  of  ICD9  codes  for 
specialties,  subspecialties,  and  consultation 

histories.  The  data  also  must  be  represented  in  a  way 
that  is  compatible  with  its  reasoning  methodology. 


Booze-Alien  Hamilton  (1999?)  proposed  to  analyze  each 
functional  requirement  and  to  generate  a  specific  set  of 
rules  for  it.  Those  rules  would  then  form  the  basis  upon 
which  consultation  and  communications  asset  allocation 
decisions  would  be  made.  Rule  generation  would  be  an 
iterative  process,  with  the  rules  being  modified  as  the 
complexity  of  the  GGTS  system  development  increases.  The 
GGTS  would  have  a  rule  manager  module,  a  rule-based  expert 
system  that  interprets  relationships  between  system 
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dynamics  and  user  priorities.  The  rule  manager  module 
would  apply  its  rules  to  determine  the  optimal  utilization 
of  available  resources  (both  communications  assets  and 
medical  personnel)  and  to  provide  direction  to  the  Network 
Bandwidth  Manager  Module  to  dynamically  allocate 
communications  assets. 

In  their  approach  to  the  CB,  the  rule  manager  module 
would  have  the  following  functions. 

1.  Evaluate  requests  against  user  and  facility  profiles, 
communications  availability,  request  priorities,  and 
other  mission-based  rules; 

2.  Provide  responses  to  requests  from  the  Consult  Manager 
Module; 

3.  Provide  recommendations  to  the  Consult  Manager  Module 
if  quality  of  service  or  level  of  service  parameters 
cannot  be  met; 

4 .  Track  network  resources  through  interface  with  the 
Network  Bandwidth  Manager  and  the  Network  Manager 
Modules; 

5.  Collect  status/topology  information  from  the  Network 
Manager  Module; 

6.  Determine  potential  connectivity  for  paths,  links, 
available  equipment,  and  satellite  footprints;  and 

Determine  connection  set-up  options  such  as  virtual 
path  /  virtual  circuit  (VP/VC) ,  bandwidth,  ATM  Adaptation 
Layer  (AAL) ,  and  Forward  Error  Correction  (FEC) . 

Regardless  of  the  system  alternative  architecture 
chosen,  the  Consultation  Broker  must  work  in  close 
conjunction  with  a  network  system  manager  (see  Figure  4)  . 
The  network  system  manager  tells  the  CB  what  types  of 
teleconsultations  are  feasible  at  a  particular  point  in 
time,  and  handles  all  administrative  functions  necessary  to 
initiate  and  sustain  the  chosen  connection  for  its 
duration.  France  Telecom  and  INTELSAT'S  successful 

development  of  telecommunication  hardware  management 
systems  using  G2  (ref  for  G2)  and  a  rule-based  approach 
support  the  contention  that  a  similar  methodology  would 
likely  be  appropriate  for  the  GGTS  network  system  manager. 
The  type  of  knowledge  necessary  to  have  an  effective 
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Consultation  Broker,  though,  is  sufficiently  different  from 
that  involved  in  a  network  system  manager  that  we  expect 
that  other  reasoning  approaches  would  be  more  effective  for 
the  CB  than  would  be  a  rule-based  methodology. 


Overview  of  GGTS  Broker  Process  for 
Teleconsulting  Services 


Figure  4 

Rules  are  an  effective  way  of  representing  and  acting 
on  knowledge  when  the  action  (the  right-hand-side)  parts  of 
the  rules  are  static  and  deterministic  results  of  the 
condition  (the  left-hand-side)  parts.  The  primary  hardware 
components  of  a  telecommunication  network  do  not  change 
frequently,  the  hardware's  behavior  is  highly  predictable, 
and  the  physical  processes  by  which  the  network  functions 
are  well  understood.  As  a  result,  a  written  procedures 
manual  could  be  constructed  to  define  the  behaviors  a 
person  should  follow  if  the  network  were  to  be  managed 
manually.  The  written  procedures  manual  should  be  adequate 
to  cover  almost  all  decisions  that  would  have  to  be  made  as 
long  as  the  network  is  functioning  normally.  A  rule-based 
system  is  the  computer-based  equivalent  of  such  a  written 
procedures  manual.  Reasoning  with  and  about  time  is  a 
complication  in  telecommunications  management,  but  G2  has  a 
proven  track  record  for  such  problem  domains,  as  mentioned 
earlier,  so  that  a  rule-based  artifact  using  that  product 
is  a  good  strategy  for  the  GGTS  network  system  manager. 
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The  Consultation  Broker's  task  of  matching  requestors 
to  consultants  requires  reasoning  in  rapidly  changing 
situations  in  which  procedures  are  subject  to  preferences 
as  well  as  firm  guidelines.  The  decision  environment 
changes  because  the  pool  of  available  consultants  changes 
frequently.  The  experience  and  expertise  of  consultants  who 
remain  in  the  system  evolve  over  time,  and  consultants' 
availability  may  change  on  a  daily  or  hourly  basis.  There 
are  advantages  to  routing  all  patients  with  similar 
problems  to  the  same  consultant,  because  the  consultant 
might  then  be  able  to  detect  patterns  of  occurrences  that 
are  problematic.  There  are  also  advantages  to  utilizing 
multiple  consultants  for  a  set  of  patients  with  similar 
problems,  so  that  more  than  one  individual  has  the 
expertise  and  experience  to  deal  with  each  type  of  problem. 
Rules  are  good  for  representing  knowledge  of  the  IF 
(conditions)  THEN  (actions)  type,  where  the  same  thing  is 
always  done  when  the  same  situation  is  encountered.  Rules 
are  not  as  good  for  knowledge  of  the  type  IF  (conditions) 
THEN  USUALLY  (actions)  UNLESS  (more  conditions)  IN  WHICH 
CASE  (actions) .  Telecommunication  network  management 
should  be  primarily  of  the  former  variety.  We  expect  that 
those  aspects  of  the  CB's  problem  that  are  based  on  medical 
considerations  are  primarily  of  the  latter;  portions  of  the 
CB's  reasoning  that  are  based  on  absolute  institutional 
directives  might  better  be  represented  by  rules. 

In  alternatives  or  situations  in  which  the 
Consultation  Broker  must  select  a  consultant,  its  task  to 
choose  the  most  appropriate  one  of  a  finite  set  of 
alternatives.  At  least  two  methodologies,  case-based 
reasoning  and  statistical  classification,  might  apply  to 
such  a  choice  decision  in  a  rapidly  changing  environment  of 
the  type  outlined  above.  Case-based  reasoning  reasons 
analogically  from  experience.  If  enough  consistent 
experiences  exist,  then  it  is  efficient  to  codify  the 
behavior  pattern  into  rules.  The  same  would  be  true  for 
principles  of  the  field.  Medical  reasoning,  though,  is 
traditionally  of  the  case-based  variety,  so  the  approach 
maps  well  onto  conventional  behaviors.  Case-based 
reasoning,  because  it  retains  all  experiences  as  opposed  to 
only  regularities  found  in  those  experiences,  has  the 
ability  to  "learn"  from  what  it  does,  if  we  consider 
remembering  to  be  the  most  fundamental  aspect  of  learning. 
Statistical  classification  also  has  the  potential  for 
resolving  the  CB's  problem,  if  sufficient  data  and  an 
adequate  data  model  could  be  found. 
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3.  CASE-BASED  CLASSIFICATION 

"A  case-based  reasoner  solves  new  problems  by  adapting 
solutions  that  were  used  to  solve  old  problems...Case-based 
reasoning  means  reasoning  from  prior  examples."  (Riesbeck 
and  Schank,  1989,  page  25)  .  Case-based  reasoning  (CBR) 
relies  on  analogy  as  its  basic  technique.  A  CBR  system 
relies  on  a  case  base  that  includes  a  relevant  set  of  past 
experiences,  each  stored  as  a  description  of  a  problem 
paired  with  the  solution  to  it.  To  solve  a  new  problem,  a 
case-based  reasoner  typically  first  retrieves  from  the  case 
base  the  case  whose  description  most  closely  matches  that 
of  the  new  problem.  It  then  extracts  the  solution  to  the 
case  retrieved  from  the  case  base,  adapts  it  if  necessary, 
and  then  adds  the  new  case  plus  its  solution  to  the  case 
base.  Smyth  and  Keane  (1998)  argue  that  the  retrieval  of 
the  most  similar  past  case  may  not  necessarily  yield  the 
one  that  is  best  suited  to  solving  a  new  problem,  and  that 
deeper  knowledge  is  required  to  determine  what  previous 
experience  is  most  relevant. 

As  a  conceptual  representation  of  human  problem 
solving,  the  CBR  approach  is  very  attractive.  It 

"remembers"  all  of  its  experiences.  Unlike  rule-based 
systems,  it  works  well  on  problem  domains  that  are  poorly 
understood,  for  which  there  is  only  weak  domain  theory,  and 
that  are  dynamic  over  time.  A  rule-based  system  provides 
answers  and  explains  its  reasoning  by  documenting  the  rules 
it  used.  A  CBR  system  provides  precedents,  and  explains  it 
reasoning  by  citing  a  precedent.  A  CBR  approach  is  thus 
attractive  for  problem  domains,  such  a  legal  reasoning,  in 
which  human  arguments  traditionally  proceed  by  citing 
precedents.  A  set  of  cases  might  be  used  by  a  discovery 
learning  system  to  extract  regularities  and  rules;  see 
Corruble  and  Ganascia  (1997)  for  an  example  of  machine 
learning  in  a  medical  context.  Kohno  et  al .  (1997) 

combined  rule-based  and  case-based  systems,  with  the  user 
and  the  CBR  system  detecting  and  correcting  mistakes  made 
by  the  rule-based  system.  Their  architecture  combines 
problem  solving  and  knowledge  acquisition,  with  the  case- 
base  serving  to  improve  the  rules . 

CBR  has  traditionally  been  applied  to  static  reasoning 
problems.  Ram  and  Santamaria  (1997)  proposed  an  extension 
of  the  approach  to  continuous  domains,  such  as  ones  in 
which  sensor  data  is  the  basis  for  reasoning.  They  were 
interested  in  robot  navigation.  Medical  problem  solving 
that  requires  tracking  information  over  time,  such  as 
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ventilator  management,  might  also  benefit  from  their 
formalization.  Spyropoulos  and  Papagounos  (1995)  evaluate 
various  AI  models  used  in  medicine,  and  discuss  their 
ethical  and  methodological  limitation. 

CBR  has  been  used  for  classification  generally,  as 
surveyed  by  McKenzie  and  Forsyth  (1995) .  Other  CBR 
applications  in  classification  include  Evans  and  Winter 
(1995)  and  Petersen  (1997)  . 

To  be  effective,  a  CBR  system  must  be  able  to  retrieve 
the  proper  past  case  from  its  knowledge  base  and  must  be 
able  to  adapt  the  retrieved  case's  solution  to  the  new 
problem  if  need  be.  The  adaptation  difficulty  can  be 
avoided  by  selecting  classification,  as  opposed  to 
synthetic,  tasks,  because  it  is  logical  to  identify  a  new 
case  with  those  cases  to  which  it  is  most  similar,  so  that 
no  adaptation  may  be  necessary.  Classification  requires 
recognition  of  features;  synthesis  requires  the  ability  to 
construct  an  artifact  from  a  specification.  Most  CBR  work 
focuses  on  classification  tasks,  an  area  in  which  there  are 
statistical  techniques  available,  such  as  discriminant 
analysis,  as  well  as  neural  network  models  and  tree- 
induction  approaches.  In  a  study  comparing  CBR,  linear 
discriminant  analysis,  and  neural  network  models  for  a 

medical  classification  task  (Musgrove  and  Davies,  1995, 

quoted  in  Watson,  1998) ,  a  CBR  model  performed  better  than 
a  neural  network  did,  and  the  neural  network  did  better 
than  linear  discriminant  analysis. 

While  adaptation  might  be  avoidable,  retrieval  is  not. 
Retrieval  of  a  case  from  a  case  base  is  similar  to 
conventional  database  retrieval,  in  that  a  set  of  critical 
features  must  be  defined  for  each  record  (case)  to  be 
included,  and  an  effective  and  efficient  algorithm  for 
searching  the  data  (case)  base  when  desired,  but  differs 
from  it  in  that  database  retrieval  returns  exact  matches 
and  CBR  retrieval  seeks  to  find  the  nearest  match.  Case 
representation,  that  is,  the  selection  of  the 

characteristics  for  indexing  cases,  is  a  particular 

difficulty  for  CBR  applications,  such  as  legal  opinions  or 
medical  records,  in  which  cases  use  natural  language. 
Representation  should  be  less  problematic  for  applications 
such  as  the  Consultation  Broker,  for  which  the  information 
in  a  case  could  be  solicited  using  a  menu  with  no  open- 
ended  entries.  Note  that  certain  case  descriptors  may  be 
measured  on  at  least  an  interval  scale,  such  as  age,  some 
on  an  ordered  categorical  scale  (ASA  classification  of  the 
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patient) ,  and  others  on  an  unordered  categorical  scale 
(ICD9  diagnostic  code(s),  doctor  requested  for  consult, 
medical  center  requested  for  consult) . 


Retrieval  of  the  most  appropriate  case  from  the  case 
base  requires  a  metric  that  can,  at  a  minimum,  for  any  new 
case  and  two  past  cases,  determine  which  of  the  two  past 
cases  is  more  similar  to  the  new  case.  The  metric  might  be 
a  weighted  combination  of  distance  measures  for  each  of  the 
descriptors  individually,  or  it  might  be  a  multivariate 
function  of  the  descriptors.  For  the  Consultation  Broker, 
the  more  difficult  descriptors  to  use  in  a  metric  are  those 
measured  on  an  unordered  categorical  scale.  Of  the  patient 
descriptors,  the  ICD9  diagnostic  codes  are  the  most 
important  categorical  information.  The  tree  structure  that 
underlies  the  ICD9  taxonomy  may  provide  a  natural  metric 
for  determining  how  similar  two  ICD9  codes  are. 

4.  CONSULTATION  BROKER  APPROACHES 

We  identified  four  distinct  approaches  to  the  way  in 
which  the  Consultation  Broker  might  determine  the  most 
desirable  match  of  consultant  to  requestor.  Regardless  of 
the  alternative  chosen,  the  Consultation  Broker  needs  the 
following  components  to  make  it  operational: 

A  list  of  all  potential  consultants  by  name  and  by 
specialty; 

A  list  of  sites  providing  consultation; 

A  link  to  a  network  manager  that  indicates  what  modes 
of  teleconsultation  are  feasible  between  a  requestor  and 
all  consultation  sites  at  the  current  time,  and  which  will 
enable  the  teleconsultation  once  its  participants  have  been 
determined; 

An  interface  to  collect  relevant  information  from  the 
requestor,  designed  to  make  that  information  as  complete 
and  as  internally  consistent  as  possible; 

An  interface  to  present  the  requestor's  information  to 
the  consultant  in  an  effective  fashion;  and 

A  time-keeping  capability  and  a  time-based  event 
trigger,  to  monitor  the  progress  of  the  teleconsultation 
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and  to  initiate  actions  necessary  to  guarantee  that  any 
mutually  desired  and  feasible  consultation  will  actually 
take  place. 

The  approaches  differ  primarily  in  the  degree  to  which 
the  computer  system  controls  the  brokering  process:  (a)  the 
"active  broker"  alternative,  (b)  the  automatic  alternative, 
(c)  the  e-board  alternative,  and  (d)  the  "yellow  pages" 
alternative . 

a.  The  Active  Broker  Alternative  (AB) 

This  alternative  uses  the  CB  solely  to  relay 
information  from  the  requestor  and  about  potential 
consultants  to  a  human,  who  is  neither  the  requestor  nor  a 
potential  consultant.  That  third  party  makes  the 
assignments,  using  the  data  provided  by  the  CB.  The 
"intelligence"  in  the  CB  under  this  alternative  is  in  the 
selection  of  information  to  be  presented  to  the  human 
assigner,  and  in  the  way  in  which  that  information  is 
presented  to  the  human  so  as  to  make  the  assigner' s 
behavior  as  efficient  and  effective  as  possible.  Under  the 
Active  Broker  alternative  (Figure  5)  ,  the  human  in  charge 
of  assignments  functions  somewhat  like  an  air-traffic 
controller,  monitoring  and  managing  the  requests  for 
teleconsultation  as  they  are  matched  up  with  consultants. 


ACTIVE  BROKER 


PHYSICIANS 


THIRD 

PARTY 
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Figure  1 

b.  The  Automatic  Alternative  (AU) 

In  the  Automatic  Alternative  (Figure  6)  ,  the 
Consultation  Broker  behaves  in  an  autocratic  fashion, 
deciding  on  a  match  without  user  intervention.  The  CB  may 
use  one  or  more  methodologies,  such  as  those  that  use  past 
experience  (for  example,  case-based  reasoning)  ,  statistical 
analysis  of  a  database  (e.g.,  classification),  or  an 
electronic  procedures  manual  (e.g.,  rule-based  reasoning) 
as  appropriate.  Independently  of  the  methodology  used,  the 
Automatic  Alternative  requires  as  inputs  knowledge  of: 

detailed  profiles  of  consultants,  and  a  list  of 
consultants  available  at  the  time  of  the  consultation, 

detailed  profiles  of  all  sites  providing  service, 

the  list  of  cases  requesting  consultations, 

a  uniform  coding  of  cases,  past  and  present,  and 

at  least  one  assignment  methodology. 


AUTOMATIC 
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CONSULT 
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Figure  6 
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c.  The  Electronic  Board  Alternative  (EB) 

Requests  for  consults  are  posted  on  an  electronic 
bulletin  board  (Figure  7),  and  the  consultants  select  the 
cases  in  which  they  want  to  participate  on  a  first-come- 
first-served  basis.  As  the  requests  are  posted,  the  CB 
alerts  all  consultants  in  the  requested  specialty  of  the 
availability  of  a  new  case.  If  a  case  is  not  selected  by 
any  consultant  within  a  time  period  that  is  a  function  of 
the  urgency  of  the  case,  the  CB  assigns  it  in  an  autocratic 
fashion,  as  it  would  in  the  automatic  alternative.  Note 
that  the  consultants  decide  on  the  case  assignment  in  this 
alternative.  The  requestor  may  ask  for  a  particular 
institution  or  a  particular  physician,  but  it  is  the 
respondent  who  determines  if  that  particular  match-up 
occurs.  The  CB's  role  in  supporting  the  matching  process 
is  limited  to  notification  to  all  potential  respondents, 
and  provision  to  them  of  the  information  from  the  requestor 
so  that  they  can  make  up  their  minds.  In  addition  to  the 
functionalities  of  the  automatic  alternative,  which  are 
needed  if  no  consultant  chooses  to  respond  to  the  request 
for  a  consult,  to  implement  the  e-board  alternative  the  CB 
requires  that: 

the  requests  must  be  disseminated  to  all  provider 
nodes  along  with  all  the  data  available  for  the 
consultation,  and 

the  blackboard  must  be  kept  in  real  time. 
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Figure  7 

d.  The  Yellow  Pages  Alternative  (YP) 

As  opposed  to  the  e-board  alternative,  in  which  the 
consultant  is  the  decision-maker,  the  "yellow  pages" 
(Figure  8)  alternative  puts  the  requestor  in  charge  of 
selecting  the  consultant.  The  CB  assists  the  requestor  by 
supplying  a  list  of  available  consultants  (the  "yellow 
pages"  listing)  together  with  background  information  that 
might  be  useful  in  making  a  selection  (the  "yellow  pages 
ads") .  As  in  the  e-board  alternative,  the  CB  would  have  to 
take  charge  and  make  a  selection  of  a  consultant  if  the 
requestor  did  not  do  so  within  an  appropriate  length  of 
time.  The  CB  must  also  be  capable  of  making  the  assignment 
if  the  requestor  asks  the  CB  to  do  so.  In  addition  to  the 
functionality  of  the  automatic  alternative,  and  a  time¬ 
keeping  functionality  as  in  the  e-board  alternative,  to 
implement  the  "yellow  pages"  alternative  the  CB  would  need: 


detailed  profiles  of  all  consultants  along  with  their 
history  of  clinical  work. 


detailed  profiles  of  all  sites  providing  service,  and 
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the  ability  to  present  the  correct  subset  of  the  above 
information  to  a  requestor  in  real-time  in  a  useful  format. 


YELLOW  PAGES 


PHYSICIANS 


Figure  8 


5.  AN  EXAMPLE  USING  A  TELEDENTAL  PROTOTYPE 

To  illustrate  how  the  GGTS  may  function  when  it  is 
eventually  fielded,  we  developed  a  functional  prototype 
using  data  from  a  dentistry  communication  system  (TDENT) 
currently  used  by  the  US  Army.  Although  the  Army's  TDENT 
system  is  purely  a  store-and-f orward  e-mail  system,  the 
data  generated  from  it  contains  useful  information  to  model 
and  explore  some  of  the  behavior  and  features  of  GGTS. 

We  used  objects  and  classes  to  represent  the 
information  contained  in  the  TDENT  records.  In  the  US  Army 
TDENT  system,  those  data  are  usually  electronically 
introduced  using  a  standard  form  (Form  513) .  The  fields  of 
the  dentistry  system  contain  patient  demographic  data, 
patient  location,  referring  physician  data  and  location, 
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medical  data  (complaint,  history,  exam  findings, 
preliminary  diagnosis  and  treatment),  specialist/consultant 
data  and  location,  dental  procedure  codes  (CDTs)  and 
diagnostic  codes  (ICD9s) .  For  example,  the  patient  in 
Figure  9  with  fictitious  first  name  (fn59)  middle  initial 
(E)  and  last  name  (ln59)  was  involved  in  two  prior 
consultations.  Thus,  the  patient  object  for  FN59  E.  LN59 
contains  links  to  the  previous  consultations,  cl637  and 
cl642.  Each  consultation  is  an  instance  of  the 

consultation  class. 


.£  Patient  Information 


Firstnsme:  fr»59 

Ml:  E 

Lastname: 


SSN: 

In59 


DOB:  fi965  | 

Military  Status: 

[20  5 

Phone  Number  j 

d'  Male  r.  Female 

Military  Rank: 

|E4  | 

H  ome  Phone  N  umber:  j" 

Patient  ID: 

|59  j 

Previous  Consults: 

cl  642 

New  Consults: 

cl  637 

_ 


Figure  9.  A  patient  instance 

Figure  10  shows  the  information  contained  in  the 
structure  of  consultation  cl637.  The  information  in  a 
consultation  is  divided  into  three  categories: 

•  patient  demographic  and  medical  information,  displayed 
in  the  upper  half  of  the  consultation  information  dialog  in 
Figure  10, 

•  specialty,  diagnostic  and  procedural  codes  displayed 
in  the  lower  left,  and 

•  specialist/consultant  information  such  as  diagnosis, 
suggested  treatment,  and  specialist  demographic  data  and 
location,  displayed  in  the  bottom  right  panel. 

Every  consultation  (known  as  an  encounter  in  the 
medical  arena)  creates  an  object  with  the  information 
displayed  in  Figure  10.  Dental  images  could  be  attached  to 
the  patient  record  if  desired. 
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.  Consult  Information 


!  Firstname:  fn59 

}  Ml:  E  j  SSH; 

\  ^59 


DOB:  ]1965 

- - 1  Military  Status: 

|20  j: 

Phone  Number: 

SEX:  M 

Military  Rank: 

OF 

l«  | 

Home  Phone  Number: 

Patient  ID: 

159  t 

Consult  ID: 

•  Chief  Complaints: 

History: 

23  Y0  AD  MALE  WITH  Hx  OF  PULPECTOMY  #14.  PAX 

REVEALS  POSSIBLE  PERF  OF  DB  ROOT  (#1 5  FILE]. 

NO  BLEEDING  FROM  DB  CANAL  EVIDENT.  PLEASE 

Images  j 

EVALAND  TREAT. 

Exam  Findings: 

;  Provisional  Diagnosis: 

POSSIBLE  PERF.  WHILE  CL  &SH 

Referring  Military  Rank: 
Referring  ID:  figg 


Referring  Firstname:  j 

Referring  Location:  jllfesheim 


Explanation  of 
Specialty: 


Explanation  of 
Diagnostic  Codes: 


Explanation  of 
Procedure  Codes: 


ID3331  -  treatment  of  root  canal  obstruction;  rron-surgical 
I  access.  •  ■  Jli 


Referring  Lastname:  j 

Date  of  Request;  17/24/90 


Diagnosis:  ^ 

^  s  . . . 

Suggested  Treatment 

Specialist  Rank: 

:  -r'-; 

Specialist  Firstname 

,J- r-  ■  •  ••  - . 


Ill  4 -Puipless  tooth  (unknown  PA  condition:  Cannot 
dx  from  PAX...un(iagio$tic  via  teledent... believe  it 
needs  to  be  enlarge  prior  to  transmitting  consult.) 
Cannot  view  file  from  film. 


| #14  -  refer  to  Wurzburg  for  #14RCT  and  posable 

pert  repair.  FYI:  If  you  have  a  pert  cal  me  on  the  _ [ 

phone  as  repairs  should  be  repaired  as  soon  as 
possible.  I  f  you  can  repair,  place  am  in  area  *  zi 


Figure  10.  Details  of  a  consultation  instance 


Each  specialist,  meaning  a  potential  consultant, 
belongs  to  the  class  "doctor"  which  contains  information 
about  the  specialist's  demographic  information  (location, 
unit  to  which  assigned,  if  appropriate)  and  involvement 
with  prior  consults  in  the  data  base  (past  consults, 
procedures  (CDTs)  and  problems  (ICD9s)  addressed  in  those 
consultations).  Figure  11  is  an  example  of  a  doctor  with 
ID#  101. 
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1 


Specialist  Information 


Firstname: 


!  Lastname: 


ID: 


101 


Specialty: 


Endodontics 


Wurzburg 


Facility :  Us_Army_HospitaF  Wurzburg 


Location: 


Assignment:  Csh-5 


Consults: 


1613 

i  -a. 

CDTs: 

D3220 

ICD9s: 

522.1  | 

1614 

D3320 

522.5 

1629 

D3330 

522.6 

1637 

D3331 

j 

1642 

D3332 

1650 

D3346 

1652 

D3347 

j 

1668 

D3348 

| 

1677 

D3352 

[ 

1678 

— 

D3410 

i 

f 

1679 

D3421 

[ 

1688 

D3426 

i 

[ 

1693 

D3430 

1 

1727 

1738 

[ 

1753 

| 

1763 

j 

1 

1869 

•\QA-i 

1 

Figure  11 .  An  example  of  a  consultant 

Doctor  101  is  an  endodontist  who  is  assigned  to  (the 
hypothetical)  unit  combat  hospital  Csh-5.  Note,  from 
Figure  11,  that  Doctor  101  is  the  person  who  handled 
consultation  cl637.  "Unit"  is  also  the  name  of  a  class, 
and  each  actual  unit  is  an  instance  of  it.  The  information 
represented  in  the  instance  for  Csh-5  is  shown  in  Figure 
12.  Csh-5  is  a  deployable  unit,  so  that  it  and/or  Doctor 
101  might  not  always  be  associated  with  the  same  brick-and- 
mortar  facility.  As  shown  in  Figure  12,  Csh-5  is  currently 
located  at  the  U.S.  Army  hospital  in  Wurzburg,  Germany. 
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Facility  Information 


Facility: 

j  CSH-5 

Service: 

j  Army 

Type:  j 

City: 

j  Wurzburg 

Latitude:  j  49,9 

Longitude:  j  -9,93333333^ 

State: 

1 

Deployment  j  Wurzburg 

Location: 

Country: 

j  USA 

Modality  Supported: 

Codel; 

1 

Code2: 

r . 

Close  Clear  Save 


Figure  12 .  An  example  of  a  unit 


Facilities  are  also  represented  as  objects  because  we  need 
to  know  things  about  them  in  order  for  the  Consult  Broker 
to  be  able  to  support  decision-making.  Each  actual 
facility  is  an  instance  of  the  "facility"  class,  and 
includes  information  such  as  its  geographical  location 
(longitude  and  latitude) ,  its  type  (clinic,  hospital,  and 
so  on)  ,  and  the  type  of  teleconsultation  modalities  it  can 
support.  Figure  5  shows  the  data  for  a  hypothetical 
hospital  at  Wurzburg.  The  modality  information  shown  in 
Figure  13  is  for  illustrative  purposes  only,  and  we  do  not 
claim  that  it  describes  the  capability  of  the  actual  U.S. 
Army  hospital  in  Wurzburg.  Other  than  latitude  and 
longitude,  the  information  about  the  specific  facility  is 
not  accurate  and  it  is  only  used  for  the  purpose  of 
illustration.  A  similar  object  exists  for  the  facility  to 
which  the  consultant  is  assigned  in  case  of  combat. 
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£  Facility  Information 


liiiiis;;- 


Inl  x| 


State: 


Faci^V:  |  US JWmy_Hosf*aVwu^u7g~" 


Service:  J  Army 

Qty: 


TVPe:  J  Hospital 


Wurzburg 


Latitude:  Longitude:  j  .9,93333333^ 


Country:  j  FRG 


Deployment  j  Wurzburg 
Location: 

Modality  Supported: 


Codel:  [  APO  AE  09244 


Code2:  |” 


Close 


Audio/Video 

Store/Forward 

Email 


Clear 


Save 


Figure  13.  An  example  of  a  medical  facility 

The  simplest  type  of  support  provided  by  the 
Consultation  Broker  is  for  it  to  function  as  a  database 
lookup  system,  similar  to  a  yellow  pages  directory.  In 
such  a  case,  the  requestor  chooses  the  "heading"  under 
which  to  "look  up"  doctors,  and  uses  to  system  to  identify 
potential  consultants,  ordered  either  by  their  proximity  or 
by  their  experience.  The  Broker  doesn't  try  to  comprehend 
the  nature  of  the  medical  problem,  but  does  collect  the 
Form  513  information  provided  by  the  requestor  so  as  to 
help  facilitate  the  consultation  process.  Figure  14  shows 
an  (rendered  appropriately  anonymous)  example  from  the 
TDENT  system.  In  Figure  14,  the  requestor  asked  for  Oral 
Surgeon  without  specifying  any  likely  procedural  codes. 
The  Broker,  in  Figure  15,  offers  a  list  of  available 
consultants,  where  "consultant"  in  this  case  means  anyone 
listed  as  being  an  oral  surgeon.  The  list  of  oral  surgeons 
in  Figure  15  is  sorted  in  order  of  increasing  (great 
circle)  current  distance  from  the  requestor's  location. 
Because  the  Broker's  knowledge  base  includes  information 
about  where  consultant's  units  are  currently  located,  it 
adjusts  the  proximity  list  if  consultants  are  deployed  away 
from  their  default  locations.  The  user  could  have  asked, 
instead,  for  the  list  to  be  sorted  by  the  number  of  prior 
consults  in  the  database.  If  the  requestor  had  specified 
procedural  codes,  the  lists  returned  by  the  Broker  would  be 
limited  to  specialists  with  experience  in  exactly  those 
codes  (if  there  is  such  data  in  the  knowledge  base)  or  from 
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those  with  experience  in  the  most  similar  codes  (if  there 
are  no  exact  matches  in  the  knowledge  base) . 


Consult  ID:  1 10000 

Chief  Complaints: 


Referring  Location;  vilseck 


Patient  presented  for  routine  exam.  TOB  use  X  20  yr 


Date  of  Request:  1 1 0/07/2000 


History: 


No  kown  trauma  to  area.  Patient  was  asked  to  quit 
smoking  nd  return  for  reevaluation. 


Exam  Findings: 


Exam  doctor  discovered  red  lesions  on  lower  lip 
adjacent  to  vermilion  border.  Reevaluation  after  1 0 
days,  lesions  still  present.  There  are  two  smooth  flat 
lesions,  app.  4mm  diameter.  Photos  taken  and 
attached. 


Provisional 

Diagnosis: 


erythroplakia 


Referrer:  Firstname: 


Lastname: 


Military  Rank: 


ID:  138 


I- 


Specially:  ^  ^ 
Diagnostic  Codes: 
Procedure  Codes:' 
Explanation, of  p 

...  '  i'Ov-V 

Specialty: 


|  Oral_Surgery 


URGENCY: 


MODALITY: 


zl 


Explanation  of  V 
Diagnostic  Codes: 

Explanation  of  i 
Procedure  Codes: 


Audio 

Video 

Audio/Video 

Store/Forward 

Email 


Close 


Clear 


Save 


Figure  14 .  A  user  request  for  an  Oral  Surgeon 
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£  LiGTS  Yellow  Pages 


^IMEiZST: ....... 


Referring  I  Vflseck  Specialty:  J  Oral_Pathology  CD**  J" 

Location: 


Service:  [Army  3  Moda*yi 

! 


('  ’^hd . II 


CDTs  by  Specialty: 


CD  DIAGNOSTICS 
CD  PREVENTIVE 
CD  RESTORATIVE 
CD  ENDODONTICS 
CD  PERIODONTICS 
CD  PROSTHODONTICS-REMOVABLE 


-=JSl2<l 


r] 


SPECIALISTS: 

Sort  By: 

SPEC 297,  Landstuhl,  (12  CCmJts) 

<•  Distance 

5PEC230,  Rota,  (1  consult) 

SPEC 366 ,  Betftesda,  (3  consults) 

C  Experience 

SPEC  156,  WRAMC,  (36  consults) 

Figure  15.  Broker  response  to  a  request  for  an  Oral 
Surgeon,  Yellow  Pages  alternative. 

Recall  that  in  Yellow  Pages  mode,  the  requestor  is  the 
decision-maker.  The  Broker  presents  its  relevant  knowledge 
to  the  requestor,  and  waits  for  the  requestor  to  choose  a 
consultant.  Once  the  requestor  does  so,  the  Broker  hands 
off  the  request  to  the  system  components  that  manage  the 
requestor-consultant  interaction.  If  the  Broker  is  to  make 
the  decision,  as  it  would  in  the  Automatic  Mode,  it  works 
its  way  down  the  list  of  specialists  in  conformity  with 
organizational  rules  and  preferences.  If  a  human  third 
party,  neither  the  requestor  nor  the  consultants,  is  to 
make  the  decision,  in  Active  Broker  mode,  the  Consultation 
Broker  presents  its  lists  to  that  human,  and  waits  for  that 
human's  decision.  Finally,  in  Electronic  Board  mode,  the 
consultants  are  the  decision-makers,  so  the  Consultation 
Broker  would  post  the  Form  513  information  to  all  relevant 
potential  consultants  and  await  a  response  from  them. 
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6.  EVALUATION  OF  ALTERNATIVES 

Independently  of  the  methodology  used  to  assign/select 
a  consultant  in  a  case,  it  is  still  necessary  to  decide 
which  is  the  configuration  of  the  Consultation  Broker.  As 
mentioned  earlier,  we  identified  four  alternative 
configurations : 


Automatic  (AU) , 

Electronic  board  (EB) , 

Yellow  pages  (YP) ,  and 
Active  broker  (AB) . 

As  in  the  majority  of  real  life  situations,  making  a 
decision  about  the  configuration  of  the  Consultation  Broker 
involves  not  just  tangible  benefits  and/or  costs  but  also 
intangibles  which  need  to  be  included  in  the  decision 
making  process.  The  measurement  of  intangibles  was  not 
possible  until  a  few  years  ago  when  Saaty  (1977,  1986) 

developed  the  Analytic  Hierarchy  Process  (AHP) .  This 
theory  is  based  on  the  use  of  relative  measurement  to 
construct  ratio  scales,  which  can  be  combined  in  hierarchic 
structures  to  yield  nonlinear  approximations.  The  theory 
is  based  on  three  principles:  decomposition  (construction 
of  a  hierarchy  or  network  of  interactions) ,  measurement 
(development  of  ratio  scales)  and  synthesis  (hierarchic  or 
network  composition.  This  theory  requires  expert  knowledge 
to  develop  the  decision  making  structure  (i.e.,  the 
hierarchy)  and  to  assess  the  relative 

importance/contribution  of  the  criteria  and  the 
alternatives  (i.e.,  ratio  scales  from  pairwise 
comparisons) .  As  an  illustration  of  how  the  alternatives 
could  be  evaluated,  we  give  below  (Figure  1)  a  simple 
hierarchy  of  objectives,  criteria  and  alternatives  that 
could  be  used  by  a  group  of  experts  to  evaluate  the  four 
alternatives.  It  is  important  to  note  that  this  theory 
does  not  have  the  same  shortcomings  as  some  other  theories 
such  as  utility  theory  or  outranking  methods.  The  AHP 
allows  for  intransitivities  of  the  decision-makers  and 
hence,  the  logic  behind  it  must  be  non-monotonic  which 
allows  for  new  facts  not  necessarily  consistent  with  all 
other  existing  facts  at  the  time  of  a  decision. 
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The  selection  of  an  alternative  could  consider  the 
following  four  objectives: 

Optimal  utilization  of  resources 

Maximum  accessibility  of  service. 

Quality  of  care,  and 

Readiness. 


These  objectives  could  in  turn  be  decomposed  as  in  Figure 
16.  Below  the  level  of  objectives  we  should  insert  a  set 
of  criteria  which  will  help  to  measure  the  contribution  of 
the  alternatives  to  the  attainment  of  the  objectives. 

To  evaluate  the  four  alternative  configurations  we 
first  need  to  assess  the  importance  of  the  objectives,  then 
the  importance  of  the  criteria  with  respect  to  each 
objective  and  finally  the  relative  importance  of  the 
alternative  configurations  with  respect  to  each  criterion. 
The  results  of  these  comparisons  can  then  be  combined  into 
a  unidimensional  scale  which  will  indicate,  in  relative 
terms,  the  importance  of  the  alternative  according  to  all 
the  objectives  and  criteria.  This  evaluation  requires 
knowledge  and  experience  on  the  alternative  configurations, 
which  is  not  yet  readily  available.  It  is  the  subject  of 
future  research.  We  propose  to  select  four  cases  to 
illustrate  the  four  configurations  and  show  their 
weaknesses  and  strengths.  A  good  example  of  a  case  similar 
to  this  one  was  summarized  in  Vlhakis  and  Partridge  (1989) . 
This  publication  is  a  summary  of  a  larger  study  conducted 
by  the  U.S.  Department  of  Energy  and  the  Office  of  the 
Inspector  General  to  determine  the  adequacy  of  security  at 
selected  U.S.  Department  of  Energy  facilities  that  produce 
nuclear  weapons.  The  AHP  provided  a  flexible  tool  for 
assessment,  planning,  and  allocation  of  resources  for 
enhanced  security. 

Note  that  to  obtain  a  synthesized  scale  for  the 
configurations  we  would  need  to  combine  the  individual 
scales  obtained  under  each  criterion  (see  for  example  Table 
1)  . 
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Desirability  of  Alternatives 


•  Comms 

•  Provider 

•  Patient 


•  Provider 

•  Patient 

•  Remote  Fac. 


•  Attending 

•  Consultation 

•  Outcome 


•  Individual 
Health  Rdns 

•  CINC’s  Mission 


Criteria 


Automatic  Electronic 
Board 


Yellow  Active 

Pages  Broker 


Figure  16.  Consultation  Broker  Alternatives  Hierarchy 


Table  1 :  Reasoning  Methodology  Rating  Matrix 
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Using  the  principle  of  hierarchic  composition,  the 
relative  standing  of  the  configurations  would  be  given  by 
the  configuration  priority  (CP) : 

CP(i )  =  X  "ijUj  +  X  aydj  +  X  ctjCj  +  X  V) 

7=1  7=1  7=1  7=1 

Resources  +  Accessibility + Quality  +  Readiness 


7.  CONCLUSIONS 

In  this  paper  we  describe  a  global  telemedicine  command 
and  control  system  (GGTS)  under  development  that  will  enable 
telemedical  consultations  to  occur  anywhere  in  the  world, 
regardless  of  location  or  transportation  medium.  The  GGTS 
consists  of  four  distinct  functional  nodes:  Field, 

Requestor,  Broker  and  Provider.  Each  one  of  these  nodes 
provides  a  specific  interface  to  the  communications  network 
that  supports  medical  operations  across  the  Operational 
Continuum.  This  paper  concentrates  on  the  Consultation 
Broker  (CB)  component,  which  is  operationalized  using 
Artificial  Intelligence  methodologies  such  as  case-based 
reasoning  and  classification  theory.  The  CB  is  an  object 
oriented  global  teleconsultation  system  designed  to  route 
incoming  requests  for  consultation  to  the  appropriate  "on 
duty"  medical  personnel.  It  may  adopt  one  of  four 
alternative  architectures  described:  automatic,  active 

broker,  yellow  pages  and  electronic  board.  We  illustrate 
the  data  structures  used  in  the  CB  and  how  these 
architectures  would  handle  different  teleconsultations  with 
data  from  a  US  Army  Teledentistry  system  in  operation. 

Many  of  the  concepts  embedded  in  the  Consult  Broker  are 
new,  and  hence  they  are  evolving  as  data  become  available. 
In  general,  there  are  "common  sense"  specifications  that  we 
would  like  the  alternative  configurations  to  satisfy.  These 
requirements  will  be  refined  once  working  prototypes  of  the 
configurations  are  available.  In  general,  the  Consultation 
Broker  architecture  should: 

(1)  Be  a  virtual  link  between  healthcare  providers  and 
medical  centers  for  24/7  Telemedicine  Consultations. 

(2)  Leverage  existing  and  available  military 
telecommunication  infrastructure  backed  up  by  civilian 
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networks  to  provide  remote  military  health  care  providers 
with  Telemedicine  access  to  consulting  physicians  at 
medical  centers. 

•  System  must  function  with  humans  to  enhance  their 
efficiency 

(3)  Provide  centralized  transparent  network  management  and 
consult  routing  with  knowledge-based  system  technology. 

•  Dynamic  and  flexible 

•  Deal  with  uncertainty 

•  Adjust  to  change 

•  Similar  to  telecommunications  routing  system  for  phone 
calls 

(4)  Have  mirrored  capability  for  Consult  Broker 
Architecture  to  ensure  that  if  the  initial  consultation 
broker  drops  off-line,  the  consult  in  progress  is 
maintained  and  new  consultations  are  rerouted  to  the 
alternative  site. 

(5)  Be  sensitive  to  requirements  of  all  potential 
customers,  who  include,  at  a  minimum,  the  patient,  the 
primary  care  giver,  the  Physician  at  the  GGTS  control 
center  and,  the  Consulting  Physician  at  the  Medical  Center. 

(6)  Be  able  to  be  responsive  to  and  incorporate  current  and 
future  policy  decisions. 

(7)  Be  continuously  aware,  along  with  Network  and  Bandwidth 
managers,  of  the  state  of  physical  resources  and  be  able  to 
respond  to  unexpected  changes  in  the  availability  of  those 
resources . 

(8)  Be  able  to  recognize  the  criticality  of  the  request 
(life  threatening)  and  have  a  time  management  capability  to 
clear  requests  and/or  upgrade  previous  requests  after  an 
elapse  of  time. 

(9)  Remain  in  standby  mode  until  initiated,  making  it  the 
triggering  mechanism  of  the  GGTS  architecture. 

(10)  Once  initiated,  evaluate  consults  predicated  upon: 

•  Patient  Information  to  include  initial  diagnosis  and 
criticality 
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•  Attending  Primary  Care  Giver  requirements 

•  Knowledge  of  physician  based  protocols 

•  Network  and  Bandwidth  resource  availability 

•  Operational  Policies 

(11)  Provide  a  medical  decision  maker  with  a  list  of 
prioritized  options  with  whom  to  initiate  the  consult. 

(12)  Augment  its  knowledge  base  with  the  results  of 
completed  consult  case  information. 
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Appendix  2: 


May,  Vargas,  Johnson,  Gilbert,  and  Illi,  "Global  Grid 
Telemedicine  System, "  presented  at  ATA  Fourth  Annual 
Meeting,  Salt  Lake  City,  Utah,  1999 


1. 


2. 


Global  Grid  Telemedicine  System 
(GGTS) 


Mr.  Jorrokl  II.  May  ^ 
Dr.  Luis  (i.  Vanias 
Dr.  Susan  Johnson 
\l\:  1  :ilvK<  iSli 

. •!  htKbiin-l! 

I  )r.  ( iarv  <  iillvrt 
:<  i 

Orio  Illi.  MSSM 

I  \  I  R( ' 


What  is  GGTS? 

The  Global  Grid  Telemedicine 
System  (GGTS)  is  a  proposed  Global 
Telemedicine  Command  and  Control 
System  (Zajtchuk  and  Zajtchuk, 

1996)  that  will  enable  telemedical 
consultations  to  occur  anywhere  in 
the  world,  regardless  of  location  or 
transportation  medium. 


3. 


4. 


WHY  IS  GGTS  NEEDED? 

•  Historically,  the  military  medical 
community  has  had  limited  access  to 
tactical  and  strategic  communication 
systems.  Increased  use  of  telemedicine 
has  previously  resulted  in  the 
employment  of  expensive,  stove-piped 
communications.  As  global  telemedicine 
operations  are  incorporated  into  doctrine, 
the  need  for  a  reliable,  scalable,  rapidly 
adaptable  virtual  network  is  becoming 
apparent. 


WHY  IS  GGTS  NEEDED? 

•  ...a  communications  network  alone 
is  not  adequate.  A  consult  routing  system 
which  considers  Command  guidance, 
availability  of  resources,  physicians’ 
preferences,  and  medically  sound 
consultation  practices  is  essential  to 
effectively  route  telemedicine  consultations 
to  the  appropriate  consultant. 
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GGTS  Concepts 


•  Provide  anlioli/ed  transparent  network 
management  and  consult  routing  with 
intelligent  decision  support  s\stem  ti'dinologv. 

•  Leverage  existing  and  available  military 
teleeommunieatioji  infrastructure  backed  up  In 
eiv  (ban  net w  oi  ks  to  prov  idc  remote  militarv 
health  care  providers  with  Telemedicine  access 
to  l  S  \-based  medical  centers. 

•  Consultation  Broker  Server  and  Network 
Manager  do  not  have  to  be  co -located  but  must 
Ih>  integrated. 


GGTS  Proposed  System  Components 


Provider  Node  j 

‘■--wji  ,\:t 


Consult  Broker  Server 
Architecture 


i;  sSSiiaiasimiamM 


Bandwidth 

MGR 

Security 

Firewall 

■■■ 

mmmm 

Consult 

MGR 

1 

Network 

MGR 

Real-time 

Knowledge-Based  Sysleni 


5. 


BOSNIA  EXPERIENCE 

•  ...providing  a  reliable  back-bone 
communications  infrastructure  is  not  enough. 
More  flexible  and  responsive  network 
management  and  prioritization  policies  and 
procedures  as  well  as  end-to-end  systems 
integration  services  and  support  are  also 
essential.” 

-  Telemedicine  Report:  Case  Study  of  Operation 
Joint  Endeavor  in  Bosnia 
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What  Knowledge  Does  GGTS 
Need? 

-  A  list  of  potential  consultants 

-  A  list  of  sites  providing  consultation 

-  A  link  to  a  network  manager  with  feasible 

modes  of  consultation 

-  An  interface  to  collect  relevant  information 

from  the  requestor 

-  A  time  keeping  capability  and 

-  A  time-based  event  trigger 


9. 


<  .<  .  I  S  (  niK  t  |)1  Hill  VlchiUcUll'C 


Put  picture  of  ii  doc  somewhere 

COTS  Broke r 


Consultation 
Ret]  ues  to  is 


-  Modality  (e.g.,  audio,  real-time  full  motion, 

sto  re-a  n  d-f  o  rwa  rd ) 

-  Patient  demographic  data  (e.g.,  name,  date, 
location,  age) 

-  Consultation  specific  data  (e.g.,  requesting 
physician’s  specialty,  consultation  urgency, 
subspecialties  requested,  type  of  equipment 
used  or  available,  history  of  present  illness,  and 
current  differential  diagnosis). 


WHAT  IS  CASE-BASED 
REASONING? 

CBR  is  [...]  reasoning  by  remembering. 

Leake.  1996 

A  case-based  reasoner  solves  new  problems 
by  adapting  solutions  that  were  used  to  solve 
old  problems. 

Riesbeck  &  Schank,  1989 


13. 


Knowledge  base  entry  for 
Doc-5991 


15. 


Consultation  Request 

Patient  demoqiaplm-'i 


Patient  Case  Base 


Patient  demographics 

Name 

Patient-1 

PaMent-2 

Dale 

04/01  /I  999 

04/01/1999 

1  oe. Df. 'll 

Bosnia 

Somalia 

A  1 1  •  ■ 

21 

34 

Consult  specific  data 

1!  equoster’s  specialty 

D  ermatology 

O  rthopedics 

Urgency 

4 

2 

S  pecia  Ity  r  eg  u  »?Vt>Ml 

Maxillofacial  surgery 

Internal  medicine 

Pinna ry  1C  D  u  rode 

802.8 

276 

Consultation  record 

M  od  a  lity 

Full-motion  video 

Audio 

C  onsu  Ha  nt 

Doc  8526 

Doc-1319 

14. 


Knowledge  base  entry  for 
Doc-8526 


16. 


EXAMPLE  TELEMEDICAL  ENVIRONMENT 

PHYSICIANS 


Patient  X 


REQUESTER 


CONSULT 

BROKER 
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17  . 


18. 


GGTS  Alternative 
Configurations 

-  Automatic  (AU) 

-  Active  Broker  (AB) 

-  Electronic  Board  (EB) 

-  Yellow  Pages  (YP) 


PHYSICIANS 


19. 


20. 


ACTIVE  BROKER  H  ELECTRONIC  BOARD 


21.  22. 


YELLOW  PAGES 


PHYSICIANS 


Criteria  for  Selecting  the  Best 
Configuration  Alternative 


Optimal  utilization  of  resources 

-  Maximum  accessibility  of  service, 

-  Quality  of  care,  and 

-  Readiness. 


(evaluation  of  Alternative 

Configurations 


Desirability  of  Alternatives 


Resources  Access  to  Care  Quality  of  Care  Readiness 


Jomms  • 

■  lovider  *  r  a::~rt 

-atient  *  Remote  Foe 


•  Attending 

•  Consultation 

•  Outcome 


•  Individual 

•  Health  Rdns 

•  CINC’s  Mission 


Automatic  Electronic  Yellow 

Board  Pages 


Active 

Broker 
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29. 


The  Active  Broker 

-  DM:  Third  party  human 

-  DOLE  of  CB:  Intelligently  support  DM 

-  OBJECTIVE:  Make  the  assignees  behavior 

as  efficient  as  possible. 

-  CONS:  Assigner  stress 

-  INPUTS:  -  profiles  of  consultants  &  sites, 

-  list  of  RFCs, 

-  audit  trail  capability 


31. 


Yellow  Pages 

-  DM:  Requester 

-  I  DOLE  of  CB:  Route  proper  consultant  info  to 

requester 

-  OBJECTIVE:  Make  right  consultants  available 
to  requester  &  make  sure  a  consult  happens 

-  CONS:  “Best"  consultant  not  guaranteed 

-  iNPUTS:  -  profiles  of  consultants  and  sites, 

-  selection  model 

-  real-time  blackboard 

-  assignment  methodology 


30. 


Electronic  Board 

-  DM:  Consultants 

-  ROLE  of  CB:  Route  requests  to  proper 

consultants 

-  OBJECTIVE:  Make  requests  available  to  the 

right  people  &  make  sure  a  consult  happen 

-  CONS:  “Best”  consultant  not  guaranteed 

-INPUTS:  -list  of  RFCs, 

-  routing  model, 

-  real-time  blackboard 

-  assignment  methodology 
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Methodologies 

-  Case-based  Reasoning 

A  case-based  reasoner  solves  new  problems  by 
adapting  solutions  that  i veto  used  to  solve  old 
problems. ..Case-based  reasoning  means  reasoning 
from  prior  examples."  (Riesbeck  and  Schank,  1989. 
page  25). 

-  Rule  Systems 

-  Classification  theory 

-  Statistical  classifiers 

-  Neural  Nets 

-  Machine  Learning 


Appendix  3: 

May,  Vargas,  Gilbert,  Illi,  Rocca,  and  Jacobs,  "GGTS 
Consult  Broker,"  presented  at  the  ATA  Fifth  Annual  Meeting, 
Phoenix,  Arizona,  2000 

1.  2. 


GGTS  Consult  Broker 

Jerrold  H.  May,  University  of  Pittsburgh 
LlliS  G.  Vargas,  University  of  Pittsburgh 
Gary  R.  Gilbert,  University  of  Pittsburgh  &  TATRC 
Orlando  J.  Illi,  U.S.  Army  MC4  Program 
Mitra  A.  Rocca,  University  of  Pittsburgh  &  TATRC 
William  G.  Jacobs,  University  of  Pittsburgh 

ATA  Phoenix,  23  Way  2000 


Structure  of  this  merged  talk 

•  Why  are  we  doing  this? 

•  Last  year  /  this  year 

•  A  flow  chart  for  the  Consult  Broker 

•  The  Consultant  Generator  &  interfaces 

•  6  examples  from  a  Tdent  prototype 

•  A  naive  classifier 

•  Where  do  we  want  to  be  next  year? 


5. 


6. 


Goals  of  this  project 

•  System  -  to  make  the  management  of 
teleconsultations  more  efficient  and 
more  effective 

•  Research  -  to  learn  how  case  based 
reasoning  can  be  combined  with 
management  science  techniques  in 
decision  support 


Last  year  /  this  year 

•  Last  year:  defined  the  scope  of  the 
problem,  figured  out  four  interface 
modes,  and  identified  probable 
methodologies 

•  This  year:  worked  on  a  teledental 
(Tdent)  prototype  using  real  data  to  see 
what  actually  works 


(,(iIS  CONSULT  BRORhR  1  UNCI  IONS 


Litres  CONSULT  BRORUR  1  UNO  HONS 


Vlcipi.u  \  Cheek  A 


Location  liener.iuir  A. 
Consistence  Check 


Consultant  (ic  iterator  I 
A  Feasibility  Check 


\klck|...Kv  Check  A 


;  I  ocation  Cone  rat  or  A 


(icner.Hiir 


t  it .  1  S  CONSULT  BRORl.R  1  UNCTIONS 


OtiTS  CONSULT  BRORUR  IU  NOTIONS 


\ilo.jiiae>  Check  A 
S|VLl.lIl>  ( icnctatoi 


Specialty  | 
C  dictator  i 

T 


Consultant  Cenctator  | 
>V  TcusibiUts  Check  ! 


CL\;uae\  Check  A 


I  oration  lieiiernior  .V 
Consistence  Check 


C  'oilsuIi.hu  Oeuer.iior 
A  Teasibilite  Check 


l  iKMiion 
Uonetator 

* 


\|amrrjtri  it*  aticn, 
■\^ca!hmr 

Nr-11*' 

\  ■' 


,v 


Does  tills  request 


CONSULTANT  GENERATOR 


CONSULTANT  GENERATOR 


Can  we  give  the  user^ 
exactly  what  he/she 
wants? 


i  !  Send  Info  to  Consultant  @  Location 
\  via  requested/available  modality 


CONSULTANT  GENERATOR 


CONSULTANT  GENERATOR 

e 


What  docs  the 
requester  want: 

11)  SELECT  from  a  list  of 
known  consultants, 
12)  ASK  whomever 
is  available 
13)  HELP. 

I  do  not 
know 


AUAlicmafe 

Syste,mfselectsa.; 
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Examples  from  a 
system  based  on 
records  of  the  Army’s 
TDENT  system 


Example  1 

Type  1  TC,  peacetime 


GGTS  Alternative 
Configurations 


YELLOW  PAGES 


PHYSICIANS 


Yellow  Pages 
Electronic  Board 
Automatic 
Active  Broker 


Requester 
Consultants 
Computer 
Third  Party 


ELECTRONIC  BOARD 

PHYSICIANS 


AUTOMATIC 


ACTIVE  BROKER 


PHYSICIANS 


PHYSICIANS 


REQUESTER 


CONSULT 

BROKER 


REQUESTER 


CONSULT 

BROKER 


THIRD 

PARTY 


Example  2 
Type  1  TC,  wartime 
(numbered  medical  groups 
may  have  been  deployed  to 
other  than  their  usual 
locations) 


Example  4 
Type  2  TC 

■xact  matches  in  case  base 


Request  is  for  a  real  time  consult  for  a  patient  who 
requires  CDTs  D4314,  D4240  and  D4263 


Example  4 

Type  2  TC 

Exact  matches  in  case  base 

D4341 , 

D4240,  D4341  D4240 

&  &  &  D4341  D4240  D4263 

ID  Location 

D4263  D4240  D4263  only  only  only 

Available 

3  Fort  Detrick 

i  s 

Now 

100  Wuerzburg 

2  4 

+16h  " 

203  Vogelweh 

1  2  1 

+2h  . . 

278  Leighton 

1  1  11 

+12h " 

378  ForEustis 

1 

Now 
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Derived  Stimulus  Configuration 
Euclidean  distance  model 


CDTs 

x  y 

D3910 

-2.2459  1.0839 

(D3460  D3470) 

-2.1906  -1.3184 

D3920 

-2.025  -1.3984 

D3450 

-1.6023  0.1843 

D3430 

-0.8711  0.5881 

D3950 

-0.4368  1.3583 

(D3410  D3421  D3425  D3426) 

-0.3692  0.4285 

D3999  . 

•0.0661  -3.5689 

(D3351  D3352  D3353) 

0.0262  0.2046 

(D3346  D3347  D3348) 

0.1406  0.1825 

D3333 

0.8989  0.3301 

(D3310  D3320  D3330) 

0.9146  0.2782 

(D 323 0D3240) 

0.9257  0.24 

(D3220  D3221) 

0.992  0.1267 

(D3110  D3120) 

1.0429  -0.3543 

D3331 

1.4935  0.2396 

D3332 

1.5465  0.0867 

u 


Example  6 
Type  3  TC 

nceriainty  in  CDT  code 


Requester  asks  for  a  periodontist,  and  specifies 
D4341,  but  isn’t  certain  of  it.  By  multidimensional 
scaling,  D4355  and  D4910  are  similar. 
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Classification  Example  1:  Single 
dominant  category 


i  nit  Classifier 


saol 


Classification  Example  2:  Two 
dominant  categories 


I  4  Ccnsuit  CLissifier 


05SIBLE  23.0769Z3 
AS  55.76623 


HAS  40  3846 17 
HAVE  36  53846 
PT  26.923077 


Classification  example  3: 

No  dominant  category 

i  C  :,i!  CLtSsit 

isn 

HHim 

ii  JAY  COMMAHERE  PLEASE  EVAL  AND  MAKE 
ij  RECOMMENDAT.ON  FOR  ADDITIONAL  SURGERY 
r  PERIODS  RE  HX  OF  FLAP  SURGERY  AND  DISTAL 
f  WEDGE  IN  APRIL  98  -  MAND  RT  PERIODHERE  QUAD 
■?  PERIODHERE  P0  6MM+  MAX  RT  COMMAHERE  MANO 
;  RT  COMMAHERE  MANO  LEFT  COMMAHERE  GRADE  1 
*  FURCATIONS  MAND  LEFT  ANO  RT  PERIOOHERE 
■I  LOCALIZED  MODERATE  ADULT  PERIODONTITIS 


ORTHODONTICS  1.774 
PERIODONTICS  1  472 
:  ENDODONTICS  1.304 
PROSTHODONTICS -0.574 
ORAL-PATHOLOGY  -1.691 
:  ORAL  MAX-FACIAL- SURGER Y  -2.284 


^  li  qiirihi  1  ; 


Ijtd^  t\  ^  C0^ID  t(  §  1 

I  ^SpicJkyi'.Orthbdootlt*  ..  .  SpitliKy:  PMockmtic*  '  V  3 

;  itgnfflcaftf  word*:  1  '  ~  *of  ifcnfflcant  word*;'  1.  ,'v  Ivi-' 

i-gaj  j  g-  <:•  -  -  ■ . ■ _ ••  , - : - : - : - - - 


/* .  Spiielilty:  Endodontic* '  ' 

;  !  :ii  of  filihlilcant  words :  2; 
£■  IPERIODONTITIS  30.76923 
HX  23.076923 


In  the  next  year  we  want  to... 

Extend  the  breadth  of  the  prototype  - 
how  representative  isTDENT  of  the 
general  medical  teleconsultation 
problem? 

Refine  the  types  of  teleconsultations  to 
best  match  the  KB&MS  tools  and  the 
interfaces  to  the  management  problem, 
user  requirements,  and  organizational 
rules 
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Appendix  4: 

Gilbert,  Illi,  Baker,  Brienza,  Rocca;  Vargas,  and  May, 
"Global  Grid  Telemedicine  System  (GGTS) , "  presented  at  the 
ATA  Fifth  Annual  Meeting,  Phoenix,  Arizona,  2000 


2. 


What  Is  GGTS? 


The  Global  Grid  Telemedicine 
System  (GGTS)  is  a  proposed  Global 
Telemedicine  Command  and  Control 
System  (Zajtchuk  and  Zajtchuk, 

'  that  will  enable  telemedical 
consultations  to  occur  anywhere  in 
the  world,  regardless  of  location  or 
transportation  medium. 


3. 


S!  WHY  IS  GGTS  NEEDED?  " 

•  Historically,  the  military  medical  community 
has  had  limited  access  to  tactical  and  strategic 
communication  systems. 

•  Increased  use  of  telemedicine  has  previously 
resulted  in  the  employment  of  expensive, 
stove-piped  communications. 

•  As  global  telemedicine  operations  are 
incorporated  into  doctrine,  the  need  for  a 
reliable,  scalable,  rapidly  adaptable  virtual 
network  is  becoming  apparent. 


4. 


BOSNIA  EXPERIENCE 


•  .  .  .providing  a  reliable  back-bone 

communications  infrastructure  is  not  enough. 
More  flexible  and  responsive  network 
management  and  prioritization  policies  and 
procedures  as  well  as  end-to-end  systems 
inlegration  services  and  support  are  also 
essential.” 

Telemedicine  Report:  Case  Study  of  Operation  Joint 
I  ■  ndeavor  in  Bosnia 


5. 


PACOM  Medical  Bandwidth  Study 


*  •  AND(HA)  rapidly  moving  to  Digital  CPR.  lint.... communications 
plans  don't  match  medical  data  movement  needs" 


•  “could  he  Inindled^uith  Eng,  02,  Intel  for  a  total  picture" 

•  “DISA  can  support  peacetime  needs:  wartime  support  questionable" 

•  ‘many  medical  systems  outside  firewall" 

•  "digital  patient  record  could  approach  Uiga-llytes"  ll'ile  Size) 

•  “ability  to  move  data  will  he  more  difficult  farther  forward" 


.fun  yv 


6. 


GGTS  Concepts 


•  Provide  centralized  transparent  network 
management  and 


*  Leverage  existing  and  available 

to  provide  remote  military 
health  care  providers  with  Telemedicine  access 
to  USA-based  medical  centers. 


y,-  3^ 
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V!  l-<  I\  1 


t  nnsiiltation  Consultation 

I’roviders  Requesters 


8. 


Feasibility  Study  (octy7-.ian9sr 


•  Military  bandwidth  used  on  space  available  basis  bucked  up 
l>\  comniercinl  bandw  idth  is  most  feasible  solution. 

•  \  I  \l  is  recommended  network  protocol. 

•  <  ,(j  I  S  requires  dynamic  and  transparent  bandwidth  allocation 
In  i m  i  en  and  among  military  and  commercial  communication 

M  v/(  m  v. 


I  onsullalion  Broker  Server  and  Network  Manager: 

-  do  not  have  to  be  eo-Ioeatcd. 

-  can  be  distributed. 

-  must  be  integrated. 


•  (  ioi'-nlt  broker  qua* *  s  multiple  /  ({/soiling  and  knowledge 

,  \ t  nlation  \cl/(  m, 


Proposed  GGTS  System  Components 


0w^Army  Medical  Department  Center  & 
^'”a  School  Letter  -  6  October  1998 


Consult  Provider  Node 


Real- lime 
Knowledge-Based  System 
Architecture 


GGTS  Feasibility  study  supports  conclusion  that  global 
telemedicine  network  is  feasible. 

No  requirement  has  been  formally  defined 

“telemedicine  remains  a  vaguely  defined  technical  capability” 

-  “theater  telemedicine  and  bandwidth  requirements  are  not 

established” 

ttocommcncls  demonsti citing  GGTS  in 

Joint  Medical  Ope; aliens- Telemedicine  (JMO-T) 

Advanced  Concepts  Technology  Demonsti ation  ( ACTD ) 

•  add;  css  ,0'nt  to  ':c medic: no  icquiremcnts  for  War  and  Operations 
othet  than  Wai  >OOTW). 

•  dctciminc  bandwdth  icqniicments. 

•  assess  signal  cc-mm/nity  support  requirements  and 
capabilities. 


Develop 
Consult  Broker 
Model 


JMO-T  ^f| 
rmplcmcrdnfion 
Plan 


Develop  Net  work 
Bandwidth  Manager 
l  ;  &  Security  Firewall 


Integrated  GGTS  . 
Prototype  Test 


GGTS  Consult  Broker  Model 
Development 

.  -  •  ■  ■  •/  -I  :  ''dlKburgh 

a  .i.  M  i;i.:; ■■•-nc-nt  Laboratory) 

*8aag§ 


M  FY  96  Completed  f; 
m  FY  99  Underway  |j| 
81  FY  00-02  Proposed 
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13. 


Tvpes  of  Information  Needed  for  a 
GGTS  Tele-medical  Consultation? 

-  Jality  (e.g.,  data,  text,  audio,  images,  store-and- 
forward,  full  motion  video,  etc.) 

-  miont  demographic  data  (e.g..  name,  date,  ssn,  sex, 
age.  unit,  location,  age.  etc) 

-  ;  msultation  specific  data  (e.g.,  requesting  physician’s 
specialty,  consultation  urgency,  subspecialties 
requested,  type  of  equipment  used  or  available,  history 
ct  present  illness,  and  current  differential  diagnosis). 


14. 


What  Knowledge  Does  GGTS 
Need? 

Knowledge  about: 

•  relevant  information  to  collect  from 

the  requestor 

•  medical  decision  making  and  consultation 

•  potential  consultants 

•  feasible  modes  of  consultation 

•  policies  and  constraints 

•  network  architecture,  network  &  bandwidth 

management,  & 

•  r  , !  ■ e-based  event  management 
and  record  keeping 


Criteria  for  Selecting  the 
Best  Configuration 
Alternative 

Optimal  utilization  of  resources 

-  Maximum  accessibility  of  service, 

-  Quality  of  care,  and 

-  Readiness. 


Evaluation  of  Alternative 

Configurations 


[Desirability  of  Alternatives! 

Ho  source's!  [Access  to  Carej  [Quality  of  Carej 

f  '  I 


[Readiness] 

I 


Com  ms  •  Provider 

*  Attending 

•  Individual 

Provider  •  Patient 

•  Consultation 

•  Health  Rdns 

•  atient  •  Remote  Fac 

•  Outcome 

•  CINC's  Mission 

r.iT .'/ac- 

Automatic  Electronic 

Yellow 

Active 

Board 

Pages 

Broker 

PHYSICIANS 


REQUESTER 


CONSULT 

BROKER 


REQUESTER 


ACTIVE  BROKER 


CONSULT 

BROKER 


PHYSICIANS 


ELECTRONIC  BOARD 


mi 


PHYSICIANS 


REQUESTER 


YELLOW  PAGES 


CONSULT 

BROKER 


PHYSICIANS 


THIRD 

PARTY 


GGTS  Transition  Strategy 


.ip|>lu  .iti<>n  Itruki-r 


•rors/uoi'voi 


Transition  from  Human  In  The  Loop 
to  Human  with  Expert  System  Support 


lecoosuit  GGTS  Broker 


Support  Service 

Joint  experiments 

implementation  and 

integration  of : 

TDC 

new  capabilities. 

*  ATM  Backbone 

providing  risk  mitigation 

•  Network  Mr! 

feedback  and  insights  to 

•  C2  Systems 

service  combat 

•  Security 

development  and 

acouisition  communities 

37. 


GGTS  Overview  Conclusions 

*  Implement  a  network  centered  method  to  leverage 
aci  esv  in  mi  I  i  tar  \  nctw  orks.  manaue  handw  idth.  and 
appropriate!)  mute  telemedicine  consults. 

*  \  aiidate  the  GGTS  "virtuaP  network  architecture 
and  infrastructure  concepts  for  all  milUarv  medical 
communications, 

*  Leverage  the  JMOT  AC  TD  to  collect  data  on  how 
medical  networks  are  managed  and  consults  are 
routed. 
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Appendix  5: 

Summary  of  naive  text  classification  experiments  on 
teledental  and  medical  data 

Naive  Text  Classifier  for  Use  in  Teledentistry 

Possibly  the  most  difficult  task  in  the  implementation 
of  an  effective  Teledentistry  system  is  the  classification 
of  the  consults  of  individual  patients.  Such 

classifications  are,  in  fact,  necessary  in  order  to  send 
patients  to  appropriate  specialists.  The  immediate  tasks 
were  to  observe  a  sample  of  past  dental  consults  and  to 
build  an  effective  classification  system  that  could 
successfully  classify  new  dental  consults. 

In  the  real-time  GGTS  Consult  Broker,  a  consult  form 
such  as  Standard  Form  513  is  to  be  completed  and  sent 
through  the  system.  In  order  to  send  the  consult  to  the 
appropriate  specialist,  an  intelligent  classifier  was 
constructed.  The  method  that  we  chose  to  use  was  that  of  a 
naive  classifier,  which  uses  rules  of  probability  and  Case- 
Based  Reasoning  (CBR)  to  determine  the  likeliness  of  a 
consult  being  in  a  certain  class. 

The  naive  classifier  that  was  implemented  uses,  as  its 
input,  the  unordered  set  of  words  contained  in  the 
consultation  report.  A  patient's  dental  problems  are 
summed  up  in  the  description  on  the  report .  The  words  used 
in  the  description  are  the  vital  clues  in  directing  the 
patient  to  a  proper  specialty.  Using  the  clues  along  with 
rules  of  probability  such  as  Simple  Bayes',  the  naive 
classifier  finds  the  most  likely  specialty  out  of  a  list  of 
possible  specialties. 

In  such  a  system,  a  number  of  past  consults  are  to  be 
used  as  a  training  set.  From  the  training  set,  the  naive 
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classifier  constructs  a  "trained  data-set,"  which  holds  the 
vital  information  on  word/category  frequencies  in  the  past 
data.  By  using  the  past  frequencies,  the  classifier  is 
able  to  decide  which  specialties  are  most  likely  associated 
with  new  consults,  which  use  many  of  the  same  words  in 
their  descriptions. 

The  actual  classification  algorithm  is  relatively 
straightforward,  and  is  flexible  enough  to  be  easily 
altered  and  improved.  The  algorithm,  in  its  simplest  form, 
is  summed  up  by  the  pseudo-code  shown  in  Figure  BJ-1. 


Figure  BJ-1 


This  algorithm  (Figure  BJ-1)  utilizes  a  variation  of 
the  Simple  Bayesian  method.  To  compute  a  score  for  each 
category  C,  the  algorithm  looks  at  the  list  of  words 


80 


describing  a  patient  D,  and,  for  each  word,  computes  the 

probability  that  the  word  belongs  to  a  consult  of  category 
C.  After  all  those  probabilities  are  summed,  the 
classifier  finally  adds  the  probability  of  an  arbitrary 
item  being  in  category  C. 

To  compute  these  probabilities,  the  classifier 
requires  a  trained  data  set  that  holds  information  on  all 
word/category  frequencies.  In  order  to  build  the  trained 
data  set,  the  naive  classifier  reads  from  a  file  that 
contains  data  from  the  items  in  its  training  set.  Each 
consult  in  the  training  file  includes  the  description 
(informal  diagnosis)  of  the  patient's  situation  and  the 

specialty  to  which  the  patient  was  later  referred.  The 
object  of  the  classifier  is  to  "read"  the  words  in  the 

description  of  a  patient's  dental  problems,  and  to  then 
send  the  patient  to  an  appropriate  specialist,  based  on  the 
specialty  to  which  the  classifier  believes  the  patient's 
diagnosis  belongs.  By  having  a  connection  between  the 

words  in  the  patient  descriptions  and  the  specialty  they 
fall  under,  the  classifier  is  able  to  store  word 
frequencies  for  each  of  its  respective  classes.  Those  word 
frequencies,  with  the  above  algorithm,  are  then  used  to 
evaluate  new  consults,  and  then  to  classify  them  to  their 
proper  specialties. 
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Frequencies  of  Wor 

ds  in  Specialties 

Oral 

Max/Facial 
Orthodontics  Surgery 

Endodontics 

Word 

Oral 

Pathology 

Prosthodontics 

Periodontics 

ATTACHED 

11.54 

0.00 

6.45 

0.00 

0.61 

1.75 

BORDER 

11.54 

0.00 

0.00 

0.00 

3.07 

0.00 

BUCCAL 

11.54 

1.09 

3.23 

2.60 

3.07 

12.28 

CHANGES 

11.54 

0.00 

0.00 

0.00 

0.61 

0.00 

CYST 

11.54 

1.09 

2.15 

0.00 

6.75 

3.51 

IMAGE 

11.54 

1.09 

0.00 

0.00 

6.75 

8.77 

IMAGES 

11.54 

2.17 

1.08 

2.60 

3.68 

1.75 

LESIONS 

11.54 

1.09 

3.23 

1.30 

3.07 

1.75 

NONCONTRIBUTORY 

11.54 

0.00 

1.08 

0.00 

0.00 

0.00 

OLD 

11.54 

5.43 

16.13 

7.79 

22.09 

0.00 

PACK 

11.54 

0.00 

0.00 

1.30 

0.61 

0.00 

PAIN 

11.54 

0.00 

3.23 

5.19 

22.09 

26.32 

PALATE 

11.54 

1.09 

0.00 

3.90 

1.84 

0.00 

POSSIBLE 

11.54 

9.78 

6.45 

7.79 

15.34 

22.81 

RADIOGRAPHS 

11.54 

0.00 

4.30 

0.00 

4.29 

8.77 

SHE 

11.54 

23.91 

16.13 

24.68 

23.93 

15.79 

SLIGHT 

11.54 

1.09 

0.00 

2.60 

6.13 

1.75 

SMOKING 

11.54 

0.00 

0.00 

1.30 

3.07 

0.00 

Figure  BJ-2 


Figure  BJ-2  shows,  for  each  of  these  specialties,  the 
percentage  of  consults  that  contain  each  given  word.  The 
naive  classifier  takes  advantage  of  the  fact  that  certain 
words  are  more  common  in  particular  specialties. 


82 


ORAL  ORAL 

PATH.  PROSTHO.  PERIO.  ORTHO.  SURGERY  ENDO. 

My 

-5.219524  -3.8169956  -3.90378  -4.514279  -3.459378  -4.369 

Size 

-3.610086  -6.3019023  -5.60853  -6.306038  -4.683153  -6.3149 

Biopsy 

-3.3736973  -6.3019023  -5.60853  -6.306038  -5.193979  -6.3149 

Lesion 

-2.8216286  -5.20329  -4.91538  -5.612891  -3.653534  -4.2354 

Period  (.) 

-2.3478441  -1.8132657  -1.82434  -1.98855  -1.242735  -2.3446 

is 

-2.8216286  -2.4100819  -2.54047  -2.455891  -1 .759992  -2.9137 

Comma  (,) 

-2.4894948  -1.9451932  -2.11202  -2.477397  -1.379936  -2.6013 

Great 

-6.318136  -4.510143  -4.91538  -6.306038  -5.193979  -5.6217 

Period  (.) 

-2.3478441  -1.8132657  -1.82434  -1.98855  -1.242735  -2.3446 

TOTAL 

-31.3498835  -34.11604  -33.2528  -37.95567  -27.80942  -37.06 

Figure  BJ-3 

Figure  BJ-3  shows  how  the  scores  are  calculated  using 
the  algorithm  in  Figure  BJ-1.  Punctuation  such  as  periods 
and  commas  can  either  be  taken  into  consideration  or 
omitted.  In  the  initial  tests,  we  took  all  punctuation 
into  consideration  when  classifying  consults.  The  sum  of 
all  the  words  in  each  consult  being  tested  were  added  up 
for  each  category.  The  classifier  returns,  as  the 

predicted  class,  the  category  with  the  highest,  or  least 
negative,  value. 


Naive  Classifier  Testing 

In  order  to  evaluate  its  performance,  testing  was  done 
on  the  naive  classifier.  The  first  step  was  to  take  the 
390  available  consults  that  were  classified  under  the 
Endodontic,  Periodontic,  Orthodontic,  and  Oral  Surgery 
specialties,  and  to  divide  them  into  a  training-set,  which 
would  be  used  to  build  the  classifier's  data-set  and  a 
test-set.  It  seemed  appropriate  to  start  off  building  an 
equally  distributed  training  set,  because  it  seems  the 
classifier's  probabilistic  nature  may  require  an  even 
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number  of  items  (consults)  for  each  specialty  that  it 
trains.  As  shown  in  Figure  BJ-4,  45  consults  in  each  of 

the  four  categories  were  used  to  build  the  classifier's 
trained  data  set,  giving  a  total  of  180  items  in  the 
training  set.  The  remaining  210  items  were  then  set  aside 
for  testing  purposes . 


#  of  Items 

#  of  Items  in  Training  Remaining  for 

Set  Testing 

Oral  Max/Facial  Surgery 
Periodontics 

Orthodontics 

Endodontics 

Total 

45  118 

45  48 

45  32 

45  12 

180  210 

Figure  BJ-4 


A  testing  program  was  written  to  process  the  test-set 
and  to  test  it  using  the  naive  classifier.  The  testing 
program  worked  by  reading  in  a  file  that  contained  the  211 
test  items.  The  format  of  the  file  was  identical  to  that 
of  the  file  used  for  building  the  training  set.  Each  line 
in  the  file  contained  the  description  of  the  patient's 
situation  and  the  specialty  that  the  patient  was  sent  to. 
As  each  item  was  read  in,  the  classifier  evaluated  the 
words  in  the  patient  description  and  calculated  scores  for 
each  of  the  possible  specialties.  The  specialty  with  the 
highest  score  was  passed  back  to  the  testing  program.  If 
the  specialty  returned  by  the  classifier  matched  the  actual 
specialty  of  the  item,  the  item  was  marked  as  successfully 
classified. 

Initial  tests  showed  that  the  classifier  successfully 
classified  around  70%  of  the  items  in  the  test-set.  Cross- 
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validation  was  also  used  to  test  the  accuracy  of  those 
results.  For  the  most  part,  the  results  stayed  around  70%. 

While  the  results  of  the  initial  tests  were 
impressive,  setting  aside  45  of  each  specialty's  items 
would  not  allow  for  sufficient  testing  for  every  specialty. 
Endodontics,  for  example,  had  a  total  of  only  57  available 
items,  which  meant  only  13  at  a  time  could  be  tested  under 
the  initial  testing  program,  which  used  45  of  the  57 
Endodontic  consults  for  the  training  set.  Therefore, 
another  method  of  testing  had  to  be  developed. 

This  time,  instead  of  separating  the  items  into  a 
training-set  and  a  test-set,  every  available  item  was  used 
to  build  the  trained  data  set.  This  data  set  contained  the 
word/class  frequencies  found  in  every  available  consult. 
As  a  result,  the  trained  data  set  no  longer  had  an  equal 
number  of  instances  for  each  specialty.  A  specialty  such 
as  Oral  Max/Facial  Surgery  had  more  than  double  the  amount 
of  items  than  Endodontics.  Whether  or  not  this  effected 
the  performance  of  the  classifier  was  yet  to  be  seen. 

Under  this  test,  we  also  decided  to  add  the 
specialties  of  Prosthodontics  and  Oral  Pathology  to 
evaluate  the  classifier's  performance  with  a  larger  set  of 
categories.  This  gave  us  a  total  of  534  consults. 

Testing  was  again  performed  by  reading  from  a  file, 
evaluating  the  words  in  the  descriptions,  and  then 
classifying  items  under  specialties.  A  few  changes, 
however,  had  to  be  made  in  order  to  run  the  tests  under 
this  new  method.  For  example,  only  one  file  was  now  needed 
for  building  the  data  set  and  performing  the  test.  Also, 
to  simulate  the  classifier's  function  of  evaluating  new  and 
unique  consults,  we  had  to  ensure  that  the  specific  items 
that  were  being  classified  were  not,  at  the  time,  actually 
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contained  in  the  data  set.  Therefore,  as  each  item  was 
being  classified,  the  testing  program  first  removed  that 
particular  item  from  the  trained  data  set.  The  method, 
shown  in  Figure  BJ-5,  tested  every  item  without  having  to 
distinguish  between  a  training-set  and  a  test-set: 


Pseudo-code  for  alternate  testing  method: 

Read  file  containing  all  534  consults 

Build  classifier's  data  set  using  the  534  items 
for  each  line  (item)  in  file 
{ 

remove  all  word/ frequency  information  in  'item' 
from  the  trained  data-set 

evaluate  'item'  and  return  its  specialty 
according  to  the  classifier 

record  the  classification 

restore  the  original  data  set  by  once  again 
adding  the  word/ frequency  information  of  'item' 
to  the  data  set 

) 

Figure  BJ-5 

Once  this  testing  algorithm  was  implemented,  the  new 
tests  were  run  on  the  data  in  the  Endodontic,  Periodontic, 
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Orthodontic,  Oral  Surgery,  Prosthodontic,  and  Oral 
Pathology  fields.  This  time,  the  program  not  only  recorded 
successful /unsuccessful  counts  for  each  category,  but  it 
also  mapped  the  results  into  an  "actual  vs.  predicted"  hash 
table,  and  produced  a  confusion  matrix  (Figure  BJ-6)  of  the 
test  results: 


Actual 

Endodontics 

Oral 

Max/ Facial 
Surgery 

Predicted 

i  Orthodontics 

Periodontics 

Prosthodontics 

Ora! 

Pathology 

Total 

Endodontics 

55 

0 

0 

0 

2 

0 

57 

Oral  Max/Facial  Surgery 

12 

68 

24 

0 

24 

35 

163 

Orthodontics 

5 

0 

58 

0 

12 

2 

77 

Periodontics 

19 

1 

5 

20 

29 

19 

93 

Prosthodontics 

13 

0 

2 

4 

71 

2 

92 

Oral  Pathology 

2 

1 

0 

0 

2 

47 

52 

Total 

106 

70 

89 

24 

140 

105 

534 

Figure  BJ-6 


•  — .  . . 

Correct 

Classifications 

Total  Tested 

Pet.  Correct 

Endodontics 

55 

57 

96% 

Oral  Max/Facial  Surgery 

68 

163 

42% 

Orthodontics 

58 

77 

75% 

Periodontics 

20 

93 

22% 

Prosthodontics 

71 

92 

77% 

Oral  Pathology 

47 

52 

90% 

Total 

319 

534 

60% 

Figure  BJ-7 


This  test  was  able  to  give  a  better  idea  of  the 
performance  of  the  classifier.  Figure  BJ-7  shows  that  most 
of  the  specialties  were  classified  successfully.  The 
notable  exceptions,  however,  were  Oral  Surgery,  which  had  a 
success  rate  of  only  42%,  and  Periodontics,  which  had  an 
even  lower  rate  of  22%.  It  was  also  interesting  to  see 
that,  when  the  classifier  made  mistakes,  Prosthodontics  was 
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a  frequent  mistake.  In  fact,  29  of  the  Periodontic 

consults  were  classified  as  Prosthodontics,  while  only  20 
of  them  were  correctly  classified  as  Periodontics. 

While  the  initial  results  were  promising,  they  left 
room  for  improvement.  In  order  to  possibly  improve  the 
classifier's  performance,  we  constructed  an  even  training- 
set,  to  balance  out  the  probabilistic  calculations  for  each 
specialty. 

Because  the  Oral  Pathology  specialty  had  the  least 
number  of  consults  associated  with  it  (52),  we  took  52 
random  consults  from  each  specialty  for  use  in  the 
training-set.  This  enabled  us  to  test  with  an  even  data 
set,  which  would  allow  for  an  even  distribution  across 
specialties.  We  then  tested  each  of  the  52  items,  using 
the  same  testing  method,  and  produced  new  results. 


Actual 

Endodontics 

Oral 

Max/Facial 

Surgery 

Predicted 

Orthodontics 

Periodontics 

Prosthodontics 

Oral 

Pathology 

Endodontics 

48 

i 

0 

0 

3 

0 

Oral  Max/Facia!  Surgery 

0 

45 

6 

0 

0 

1 

Orthodontics 

1 

9 

37 

0 

5 

0 

Periodontics 

4 

0 

2 

27 

15 

4 

Prosthodontics 

2 

1 

6 

8 

35 

0 

Oral  Pathology 

2 

5 

0 

0 

1 

44 

Figure  BJ-8 


Correct 

Classifications 

Total  Tested 

Pet.  Correct 

Endodontics 

48 

52 

92% 

Oral  Max/Facial  Surgery 

45 

52 

87% 

Orthodontics 

37 

52 

71% 

Periodontics 

27 

52 

52% 

Prosthodontics 

35 

52 

67% 

Oral  Pathology 

44 

52 

85% 

Total 

236 

312 

76% 

Figure  BJ-9 
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Figures  BJ-8  and  BJ-9  show  that  the  evenly  distributed 
training-set  produced  significantly  more  successful 
results.  This  has  to  do  with  the  probabilities  of  word- 
frequencies  being  evened  out  for  each  specialty.  It  is 
interesting  to  note  that  27  out  of  52  (52%)  of  the 
Periodontics  items  were  classified  correctly  under  this 
method,  while  only  20  out  of  93  (22%)  were  classified 
correctly  with  the  unevenly  distributed  training-set. 


Improvement 

Endodontics 

Oral  Max/Facial  Surgery 
Orhodontics 

Periodontics 

Prosthodontics 

Oral-Pathology 

-4.18% 
+44.82% 
-4.17% 
+30.41  % 
-9.86% 
-5.76% 

OVERALL 

+15.90% 

Figure  BJ-10 


Figure  BJ-10  shows  the  difference  (in  actual 
percentage  values)  between  the  results  of  the  unevenly- 
distributed  and  evenly  distributed  data  sets.  A  specialty 
with  a  positive  (+)  difference  indicates  an  improvement 
caused  by  the  evenly  distributed  set.  While  only  2  of  the 
6  specialties  saw  improvements  with  the  evenly  distributed 
data  set,  the  overall  performance  was  a  substantial  15.9% 
better . 

Relative  Scoring 

In  order  to  further  evaluate  and  possibly  improve  the 
performance  of  the  classifier,  changes  were  made  which 
allowed  the  classifier  to  keep  track  of  the  scores  of  each 
category  it  tested.  This  not  only  allows  for  further 
evaluation  of  results,  but  it  also  allows  the  classifier  to 
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produce  confidence  levels. 

Once  the  scores  for  each  specialty  are  calculated,  the 
classifier  gives  each  specialty  a  relative  score,  which 
shows  the  likelihood  that  a  patient's  description  falls 
under  a  specific  specialty,  relative  to  other  categories. 
Confidence  values,  therefore,  can  be  generated  from  these 
relative  scores. 

Tests  were  then  run  to  measure  the  effectiveness  and 
significance  of  the  new  relative  scoring  system. 
Distinctions  were  made  which  divided  classified  consults 
into  two  separate  categories.  A  classified  consult  was 
considered  "unsure"  if  none  of  the  specialties  had  a 
relative  score  of  over  10.  Consults  with  a  single  dominant 
specialty  with  a  relative  score  greater  than  10  were 
considered  "confident."  The  classifier  was  then  altered  to 
return  only  the  dominant  category  for  confident  consults, 
while  returning  the  entire  list  of  relative  scores  in 
unsure  instances. 

A  set  of  210  Endodontic,  Periodontic,  Orthodontic,  and 
Oral  Max/Facial  Surgery  items  were  tested  to  measure  how 
many  of  the  items  would  be  confidently  classified,  and,  in 
turn,  how  many  of  the  confidently  classified  items  were,  in 
fact,  correctly  classified.  These  confidence  levels,  if 
valid,  improve  the  classifier  by  allowing  for  distinctions 
between  definite  classifications  and  unsure  classifications 
that  may  require  further  evaluation. 

As  shown  in  Figure  BJ-11,  out  of  the  210  items  that 
were  tested,  slightly  over  half  of  them  returned  just  one 
specialty  and  were  considered  confident : 


Items  Classified 


Items  Considered 

Confident 


210  109(52%) 


Items  Considered 

Unsure 

101  (48%) 


Figure  BJ-11 

How  accurate  are  the  confident  items?  Figure  BJ-12 
shows  that  out  of  the  109  items  considered  confident,  most 
every  one  was  classified  correctly: 


Consult  Classifier 


TOOTH  COLONHERE  OCTOTHORPEHERE  19  BROKE 
TEMPORARY  HAD  ENDO  DONE  WHILE  DEPLOYED 
BY  A  FOREIGN  DENTIST  OF  SOME  SORT 
PERIODHERE  IT  WAS  INADEQUACY  DONE 
PERIODHERE  SHE  IS  7  MONTHS  PREGNANT 
COMMAHERE  OTHER  THAN  THAT  HER  HEALTH 
HISTORY  IS  INSIGNIFICANT  PERIODHERE  I  WOULD 
APPRECIATE  IT  IF  YOU  COULO  LOOK  AT  THIS 
PATIENT  PERIODHERE  I  NOW  HAVE  MY  DIGITAL 
O.AMFB  AHFfiE  ANO-CnULCSENn  J^iLA.PiCTURF 

I' '  »>|t  SuWbftr 


NDODONTICS  33.967274 
SROSTHODONTICS  -2.1129131 
ORAL-PATHOLOGY  -6.6821814 
P^IODONTICS  -7.760179 
OR^ODONTICS  -8.217321 
ORAL-MAX -FACIAL-SURGERY  -9. 194683 


lllEMM 


ON  51.086956 
HAVE  39. 130436 
PATIENT  33.695652 
THIS  35.869564 
YOU  42.391304 
HAH  34  782608 


ENDODONTICS  33.967274 
PROSTHODONTICS  -2.1129131 
ORAL-PATHOLOGY  -6.6821814 
PERIODONTICS  -7.760179 
ORTHODONTICS  -8.217321 
ORAL-MAX--FACI AL-SURGERY  -9. 194683 


Figure  BJ-13 


Figure  BJ-13  displays  a  screenshot  of  the  Consult 
Classifier,  which  allows  the  user  to  scroll  through  past 
consults  or  enter  new  consults  to  be  classified.  The 
relative  scores  of  each  of  the  categories  are  displayed  in 
the  top  right  box.  In  the  case  of  Figure  BJ-13, 
Endodontics  is  the  obvious  category.  Therefore,  Consult 


91 


#2420  can  be  classified  as  Endodontics  with  a  high  degree 
of  confidence. 

SURGEON  FINDER 

In  order  to  successfully  route  patients  to  an 
appropriate  surgeon,  we  developed  a  system  that  takes 
advantage  of  the  hierarchical  structure  of  the  ICD9  codes. 

The  Surgeon  Finder  software  we  developed  allows  a  user 
to  enter  an  ICD9  code  (diagnosis)  in  order  to  find  a 
surgeon  who  has  dealt  with  patients  with  similar  or 
identical  diagnoses.  By  using  the  data  collected  from  two 
hospitals  (UW  and  HMC)  ,  records  were  made  for  each  of  the 
338  surgeons  who  had  performed  procedures  in  either 
hospital.  These  records  included  information  such  as 
frequencies  in  which  surgeons  saw  patients  with  specific 
ICD9s. 

By  utilizing  these  records,  we  were  able  to  list 
appropriate  surgeons  for  the  user,  given  a  selected  ICD9 
code.  Not  only  could  the  Surgeon  Finder  find  surgeons  who 
have  dealt  with  patients  with  the  exact  same  diagnoses,  it 
could  also  handle  cases  where  few  or  no  surgeons  exist  with 
such  experience.  In  other  words,  if  no  surgeon  has  ever 
dealt  with  a  specific  ICD9  diagnosis,  the  Surgeon  Finder 
allows  the  user  to  "back  up"  to  a  more  general  level,  and 
find  surgeons  who  have  dealt  with  patients  with  similar 
diagnoses . 

The  hierarchical  structure  of  the  ICD9  codes  allows  us 
to  logically  move  up  and  down  to  either  more  or  less 
specific  levels  of  diagnoses.  For  example,  if  Patient  A  is 
diagnosed  with  an  ICD9  code  of  414.11,  the  user  would  enter 
the  most  specific  version  of  the  code.  In  return,  the 
Surgeon  Finder  lists  surgeons  who  have  dealt  with  other 


414.11  patients,  and  displays  the  number  of  times  they  have 
done  so: 


Figure  BJ-14 


In  this  case  (Figure  BJ-14),  no  surgeons  who  have 
treated  an  Aneurysm  (arteriovenous)  of  coronary  vessels 
(414.11)  are  available.  This  raises  the  question  of  what 
to  do  in  these  types  of  cases  where  no  exact  matches  exist. 
Our  next  step  was  to  evaluate  the  structure  of  the  ICD9 
codes  to  figure  out  the  best  way  to  handle  these  cases. 
The  ICD9  code  book  is  structured  in  a  hierarchical  fashion, 
and  the  knowledge  structure  inherent  in  that  hierarchy 
might  be  a  promising  way  to  "back  up"  and  find  surgeons  at 
more  general  levels. 
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Figure  BJ-15 

As  shown  in  Figure  BJ-15,  the  last  "1"  in  code  number 
414.11  takes  414.1  to  a  more  specific  level.  In  other 
words,  414.10,  414.11,  and  414.19  are  all  siblings  under 

the  more  general  grouping  of  414.1  codes.  Furthermore, 
414.0,  414.1,  414.8,  and  414.9  are  siblings  and  are  all 

part  of  the  general  group  of  414  codes.  This  structure  is 
what  allows  us  to  "back  up"  to  less  specific  levels. 

In  order  for  a  user  to  find  a  surgeon  who  is 
experienced  with  patients  similar  to  a  414.11  patient,  it 
is  possible  to  move  up  in  the  tree,  and  find  surgeons 
experienced  with  Aneurysm  of  heart  (414.1)  patients  more 
generally: 
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Figure  BJ-16 

In  this  case  (Figure  BJ-16)  ,  there  are  still  very  few 
surgeons  who  have  dealt  with  the  even  more  general  cases  of 
Aneurysm  of  heart.  Therefore,  the  next  logical  step  would 
be  to  move  another  level  up  in  the  hierarchy  to  look  at  all 
surgeons  who  have  treated  any  414  patients: 
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Figure  BJ-17 

As  displayed  in  Figure  BJ-17,  a  significant  number  of 
surgeons  have  had  experience  with  forms  of  chronic  ischemic 
heart  disease  classified  as  other  (414  codes  at  their  base- 
number  level) .  At  this  level,  significantly  more  surgeons 
are  available  for  selection.  The  Surgeon  Finder,  at  all 
levels,  also  allows  the  user  to  see  a  list  of  ICD9  codes 
that  a  particular  surgeon  has  dealt  with. 

If  the  user  continued  to  move  up  in  the  ICD9 
hierarchy,  the  Surgeon  Finder  would  then  move  from  the 
base-number  level  up  to  the  next  level,  which  is  a  range  of 
base  numbers  under  the  heading.  Ischemic  Heart  Disease  (ICD 
codes  410-414) .  Figure  BJ-18  shows  the  results  of  moving 
up  one  more  level . 
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Figure  BJ-18 


Surgeon  Categorization 

In  order  to  route  patients  to  appropriate  surgeons,  a 
categorization  of  the  surgeons  was  required.  The  Surgeon 
Finder  software  was  quite  effective  in  guiding  patients  to 
surgeons  who  were  experienced  with  other  similar  patients. 
However  the  question  as  to  how  far  a  patient  must  travel  to 
find  a  qualified  surgeon  remains. 

A  helpful  tool  in  deciding  which  surgeons  are 
appropriate  is  surgeon  categorization.  The  task  was  to 
separate  surgeons  into  three  groups:  Specialists ,  who 
specialized  in  a  few  types  of  patients;  Sub-specialists, 
who  specialized  in  even  fewer  types  of  patients;  and 
Generalists ,  who  deal  with  many  different  patients  with 
many  different  diagnoses. 
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The  categorizations  are  based  on  the  Hirschman- 
Herfinadahl  Index  (HHI)  for  each  surgeon.  The  HHI  is  a 
standard  concentration  measure  used  by  economists  and 
lawyers  for  quantifying  the  degree  of  monopoly  in  a  market. 
The  larger  the  value  of  the  HHI,  the  more  economic  power  is 
concentrated  on  a  small  number  of  firms.  See  William  G. 
Shepherd,  The  Economics  of  Industrial  Organization,  Third 
Edition,  Prentice-Hall:  Englewood  Cliffs,  NJ,  1990,  for  a 
discussion  of  the  HHI,  alternative  measures  of 
concentration,  and  their  use  in  economics.  If  we  consider 
surgeons  analogous  to  industries,  and  the  observed  ICD9s  as 
analogous  to  firms  in  an  industry,  then  the  HHI  might  be  a 
useful  measure  of  the  degree  of  surgeon  specialization. 
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Surgeon 

HHI 

#  of  Consults 

2847 

1307.52 

37 

2852 

393.82 

732 

2855 

5917.97 

32 

2856 

461 .07 

279 

2865 

1124.26 

13 

2869 

661.16 

22 

2871 

656.99 

103 

2924 

340.25 

102 

2945 

2800.00 

5 

2947 

3333.33 

3 

2972 

2163.05 

39 

2985 

188.02 

220 

2994 

453.51 

42 

3018 

5000.00 

2 

3026 

10000.00 

1 

3031 

833.33 

12 

3062 

4575.00 

40 

3114 

10000.00 

4 

3147 

1083.68 

27 

3148 

341.86 

59 

3368 

864.20 

18 

3372 

1851 .85 

9 

3375 

2098.77 

9 

3390 

689.06 

76 

3399 

2500.00 

4 

Figure  BJ-19 


Using  surgeons  in  the  place  of  markets  and  ICD9 
instances  in  the  place  of  firms,  HHI  values  were  calculated 
for  each  surgeon.  Figure  BJ-19  shows  a  subset  of  all 
surgeons '  HHI  scores . 

Surgeons  were  then  grouped  into  the  three  separate 
categories  based  on  their  ICD9 -based  HHI  scores.  Surgeons 
with  HHI  scores  in  the  top  25th  percentile  were  considered 
sub-specialists.  Surgeons  between  the  50th  and  75th 
percentiles  were  considered  specialists .  All  other 

surgeons,  with  HHI  scores  in  the  bottom  half  of  the 
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population,  were  considered  generalists .  Figure  BJ-20 
shows  some  of  the  groupings . 


Surgeon 

HHI 

#  of  Consults 

Category  (overall) 

1150 

5243.76 

29 

Sub-Specialist 

1339 

1090.12 

199 

Specialist 

1352 

1065.76 

21 

Specialist 

1357 

2048.61 

24 

Sub-Specialist 

1358 

485.10 

356 

Generalist 

1361 

1155.02 

170 

Specialist 

1479 

1059.01 

653 

Specialist 

1497 

343.78 

1015 

Generalist 

1521 

291.11 

1607 

Generalist 

1524 

196.88 

181 

Generalist 

1532 

1083.74 

560 

Specialist 

1635 

239.95 

303 

Generalist 

1686 

1017.45 

1452 

Specialist 

1731 

278.77 

698 

Generalist 

1735 

455.81 

897 

Generalist 

1739 

2555.65 

541 

Sub-Specialist 

1792 

1117.86 

119 

Specialist 

1797 

1191.41 

32 

Specialist 

1802 

780.53 

66 

Specialist 

1807 

100.80 

909 

Generalist 

1839 

321.32 

744 

Generalist 

2002 

4648.53 

..  ."  21 

Sub-Specialist 

Figure  BJ- 

20 

After 

all  ICD9 

HHI  scores 

were  calculated 

surgeons  were  categorized,  we  further  evaluated  the 
validity  of  this  categorization  method.  Lists  and  tables 
were  constructed  which  listed  surgeons  who  had  worked  at 
UW,  HMC,  or  at  both  hospitals: 
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HHIs  According  to  ICD9  Cod 

es 

Both  Hospitals 

HMC 

UW 

Surgeon 

HHI 

Category 

HHI 

Category 

HHI 

Category 

Match? 

6930 

305.76 

Generalist 

307.14 

Generalist 

1 

6960 

1280.28 

Specialist 

1280.28 

Specialist 

1 

6961 

3419.80 

Sub-Specialist 

2396.69 

Sub-Specialist 

3561.91 

Sub-Specialist 

1 

6983 

598.30 

Generalist 

598.3 

Generalist 

1 

6995 

200.36 

Generalist 

200.36 

Generalist 

1 

7000 

909.09 

Specialist 

1172.84 

Specialist 

1 

7001 

809.32 

Specialist 

809.32 

Specialist 

1 

7012 

347.16 

Generalist 

644.53 

Generalist 

648.42 

Generalist 

1 

7020 

769.02 

Specialist 

1753.44 

Sub-Specialist 

859.18 

Specialist 

0 

7030 

1400.00 

Specialist 

1 

7052 

1124.26 

Specialist 

1124.26 

Specialist 

1 

7056 

147.12 

Generalist 

127.92 

Generalist 

378.6 

Generalist 

1 

7093 

893.99 

Specialist 

1488 

Sub-Specialist 

907.35 

Specialist 

0 

7099 

418.88 

Generalist 

414.88 

Generalist 

i 

7163 

588.43 

Generalist 

687.19 

Generalist 

i 

7164 

2789.78 

Sub-Specialist 

2789.78 

Sub-Specialist 

i 

7270 

214.16 

Generalist 

769.23 

Specialist 

218.78 

Generalist 

0 

7272 

571.66 

Generalist 

601.02 

Generalist 

834.57 

Specialist 

0 

7276 

1984.39 

Sub-Specialist 

2017.06 

Sub-Specialist 

i 

373.43 

Generalist 

1000 

Specialist 

476.19 

Generalist 

0 

Figure  BJ-21 

Figure  BJ-21  shows  some  cases  where  surgeons  have 
different  types  of  scores  in  different  hospitals.  However, 
in  most  cases,  surgeons'  HHI  scores  were  very  similar  in 
both  hospitals. 


Surgeon  Categorization  according  to  CPT-based  HHI  scores 

Up  to  this  point,  we've  calculated  HHI  scores  and 
categorized  surgeons  based  on  ICD9  codes.  These  ICD9 -based 
HHI  calculations  indicate  how  specialized  surgeons  are  at 
treating  patients  with  particular  diagnoses.  However,  it 
could  also  be  helpful  to  know  how  specialized  surgeons  are 
at  performing  particular  procedures  {CPTs) .  On  one  hand, 
we  can  classify  surgeons  according  to  patient  problems 
(ICD9s) ;  on  the  other  hand,  we  can  classify  surgeons 
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according  to  the  solutions ,  or  the  procedures  (CPTs)  used 
to  treat  the  problems . 

CPT-based  HHI  scores  were  calculated  for  each  of  the 
surgeons  who  had  performed  procedures  at  either  or  both 
hospitals  (UW  and  HMC)  .  Once  again,  the  surgeons  with  HHI 
scores  above  the  75th  percentile  were  considered  sub¬ 
specialists.  Surgeons  between  the  50th  and  75th  percentile 
were  then  considered  specialists;  other  surgeons  under  the 
50th  percentile  were  then  again  referred  to  as  generalists . 

Tables  were  again  made  to  evaluate  the  surgeons'  new 
CPT-based  HHI  scores .  The  new  CPT-based  tables  looked  very 
similar  to  the  ICD9 -based  tables.  Most  of  the  surgeons  who 
had  performed  procedures  in  both  hospitals  had  similar  HHI 
scores  for  each  hospital.  Most  surgeons  who  were 
generalists  in  their  UW  practices  were  also  generalists  in 
their  HMC  practices.  The  same  went  for  sub-specialists. 
In  some  cases,  a  surgeon  would  be  either  a  generalist  or  a 
sub-specialist  at  one  hospital,  but  be  a  specialist  at  the 
other  hospital;  however,  the  categorizations  almost  never 
jumped  over  the  intermediate  (specialist)  group.  Figure 
BJ-21  shows  part  of  the  table  that  was  constructed. 
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HHIs  According  to  CPT  Codes 

Both  Hospitals 

HMC 

UW 

Surgeon 

HHI 

Category 

HHI 

Category 

HHI 

Category 

Match? 

6930 

413.47 

Generalist 

415.25 

Generalist 

1 

6960 

1211.07 

Specialist 

1211.07 

Specialist 

1 

6961 

776.92 

Specialist 

1074.38 

Specialist 

808.90 

Specialist 

1 

6983 

902.10 

Specialist 

902.1 

Specialist 

1 

6995 

621.90 

Specialist 

621.9 

Specialist 

1 

7000 

1322.31 

Sub-Specialist 

1666.67 

Sub-Specialist 

1 

7001 

4127.06 

Sub-Specialist 

4127.06 

Sub-Specialist 

1 

7012 

257.08 

Generalist 

468.75 

Generalist 

505.65 

Generalist 

1 

7020 

1344.03 

Sub-Specialist 

1386.92 

Sub-Specialist 

1872.45 

Sub-Specialist 

1 

7030 

1200.00 

Specialist 

7052 

2071.01 

Sub-Specialist 

2071.01 

Sub-Specialist 

1 

7056 

199.53 

Generalist 

213.07 

Generalist 

291.56 

Generalist 

1 

7093 

672.05 

Specialist 

1200 

Specialist 

649.69 

Specialist 

1 

7099 

337.57 

Generalist 

339.33 

Generalist 

1 

7163 

51405 

Generalist 

583.28 

Generalist 

1 

7164 

855.62 

Specialist 

855.62 

Specialist 

1 

7270 

130.78 

Generalist 

769.23 

Specialist 

132.63 

Generalist 

0 

7272 

226  49 

Generalist 

233.99 

Generalist 

350.62 

Generalist 

1 

7276 

2104.50 

Sub-Specialist 

2139.26 

Sub-Specialist 

1 

7300 

294.61 

Generalist 

1000 

Specialist 

335.10 

Generalist 

1 

Figure  BJ-22 


Similarities  and  Differences  between  ICD9 -based  and  CPT- 
based  Categorizations 

The  next  task  was  to  find  out  if  a  surgeon  who  is  a 
sub-specialist  due  to  his  ICD9  (diagnosis) -based  HHI  score, 
is  also  a  sub-specialist  due  to  his  CPT  (procedure) -based 
HHI  score.  In  other  words,  we  had  to  find  out  if  there 
were  a  significant  number  of  surgeons  who  specialized  in  a 
particular  type  of  diagnosis,  but  performed  a  broad  range 
of  procedures;  and  vice  versa. 

In  order  to  evaluate  this  concept,  a  confusion  matrix 
was  built  that  contained  information  on  surgeon- 
classifications.  It  compared  the  surgeon-classifications 
decided  by  ICD9-based  HHI  scores  and  the  surgeon- 
classifications  decided  by  CPT-based  HHI  scores: 
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HMC  and  UW  Surgeons 

Surgeon  groups  (Generalist,  Sub-Specialist,  Specialist)  based  on  diagnosis  (ICD9s)  compared  to 
Surgeon  groups  based  on  medical  procedures  (CPTs) 

(Surgeon  groups  based  on  HHI  percentiles  from  both  HMC  and  UW) 
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61 

120 

60 

61 
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Figure  BJ-23 

Figure  BJ-23  shows  that  61%  of  ICD9  sub-specialists 
are  also  CPT  sub-specialists ;  also,  most  of  the  other  39% 
are  classified  as  specialists .  Only  in  7  of  the  241  cases 
are  there  surgeons  with  both  sub-specialist  and  generalist 
classifications.  This  shows  that  there  is  a  connection 
between  ICD9-based  classification  and  CPT-based 
classification.  Similar  confusion  matrices  were 
constructed  that  made  the  same  distinctions  for  each 
hospital  separately  (Figures  BJ-24  and  BJ-25) 


UW  Surgeons 


Surgeon  groups  (Generalist,  Specialist,  Sub-Specialist)  based  on  diagnosis  (ICD9s)  compared  to 
Surgeon  groups  based  on  medical  procedures  (CPTs) 

(Surgeon  groups  based  on  HHI  percentiles  from  both  HMC  and  UW) 


Figure  BJ-24 
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HMC  Surgeons 

Surgeon  groups  (Generalist,  Specialist,  Sub-Specialist)  based  on  diagnosis  (ICD9s)  compared  to 
Surgeon  groups  based  on  medical  procedures  (CPTs) 

(Surgeon  groups  based  on  HHI  percentiles  from  both  HMC  and  UW) 
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Figure  BJ-25 


Specialization  of  Hospitals 

Surgeon  categorization  also  allowed  us  to  evaluate  the 
level  of  specialization  in  the  two  separate  hospitals  (UW 
and  HMC) : 


Groupings  based  on  ICD9s 
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25% 

25% 

HMC 

53% 

29% 

18% 

UW 

38% 

31% 

32% 

Groupings  based  on  CPTs 

Generalists 

Specialists 

Sub-Specialists 
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25% 

25% 

HMC 

49% 

35% 

16% 

UW 

43% 

31% 

26% 

Figure  BJ-26 

These  results  (Figure  BJ-2  6)  show  that  more  of  the 
surgeons  at  UW  would  be  considered  specialists  than  at  HMC. 
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UW  is  a  tertiary  care  University  medical  center,  and  HMC  is 
primarily  a  trauma  center,  so  the  numerical  results  are 
consistent  with  our  expectations.  Also,  note  that  the 
results  of  ICD9-based  groupings  are  very  similar  to  those 
of  CPT-based  results. 

Using  Surgeon-Classification  in  the  Surgeon-Finder  Software 

Now  that  we  had  developed  a  mechanism  to  assign  the 
classifications  "sub-specialist,"  "specialist,"  and 
"generalist, "  we  proceeded  to  integrate  those 
classifications  into  the  Surgeon  Finder  program.  The 
original  Surgeon  Finder  program  allowed  the  user  to  enter 
an  ICD9  code,  and  in  returned  a  list  of  surgeons  who  were 
experienced  with  treating  that  ICD9  code  or  ICD9  codes 
similar  to  it.  The  original  program,  however,  displayed 
only  the  number  of  times  a  surgeon  was  referred  to.  It  did 
not  take  into  account  whether  or  not  the  surgeon  was 
specialized  in  that  field,  or  whether  he  was  simply 
experienced  in  all  fields,  including  that  field.  This 
information  could  prove  to  be  vital  in  situations  where 
sub-specialists  are  required.  Also,  the  information  can  be 
helpful  in  situations  where  a  patient  does  not  need  a  sub¬ 
specialist,  and  instead  simply  wants  the  closest  available 
general  surgeon. 

Prior  to  the  HHI-based  categorization  of  surgeons,  the 
Surgeon  Finder  displayed  every  surgeon  regardless  of  his 
specialization,  or  lack  thereof.  Integrating  the  idea  of 
surgeon  categorization,  the  new  Surgeon  Finder  allows  the 
user  to  select  a  desired  scope  of  surgeons: 
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Figure  BJ-27 

Figure  BJ-27  displays  the  user's  new  options  with  the 
new  Surgeon  Finder  software.  Not  only  can  the  user  now 
observe  and  sort  surgeons  based  on  their  HHI  values  (degree 
of  specialty)  ,  but  the  user  can  also  choose  whether  to 
filter  out  certain  classifications  of  surgeons: 
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Figure  BJ-28 

Figure  BJ-28  displays  an  instance  where  the  user 
desired  only  sub-specialists.  By  having  only  the  "Sub- 
Specialist"  box  selected,  the  Surgeon  Finder  returns  only 
Sub-Specialists  who  have  dealt  with  similar  diagnoses. 

Naive  Classifier  For  Use  in  General  Medicine  (ICD9 
Classifier) 

The  naive  classifier  already  produced  successful 
results  in  Dentistry  fields.  However,  it  had  yet  to  been 
tested  under  more  general  fields  of  medicine.  Using  the 
statistics  from  the  UW  and  HMC  information,  a  naive 
classifier  was  constructed  that  classified  general  medical 
consults . 
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Figure  BJ-29 


The  naive  ICD9  Classifier  allows  a  user  to  enter  ICD9 
codes  as  well  as  other  patient  descriptions.  The  consult 
is  classified  by  using  the  list  of  words  found  in  the 
descriptions  of  ICD9  codes,  as  well  as  the  additional  text. 
In  Figure  BJ-29,  an  obvious  Cardiovascular  consult  is 
categorized.  A  classification  with  a  relative  score  as 
high  as  80.75  assures  that,  out  of  the  given  categories, 
the  chosen  category  is  almost  certainly  the  correct 
category.  However,  when  the  ICD9  codes  and  other 

descriptions  are  more  ambiguous,  the  relative  classifier 
scores  will  be  much  closer. 
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Figure  BJ-30 

In  Figure  BJ-30,  the  classifier  is  given  a  very 
ambiguous  diagnosis.  Although  the  three  ICD9  codes  and  the 
additional  commentary  are  very  different  from  each  other, 
the  ICD9  Classifier  is  still  able  to  return  the  most  likely 
specialty,  based  only  on  the  actual  words  in  the 
descriptions . 


Testing  on  Naive  Classifier  in  General  Medicine 

Tests  were  run  in  order  to  evaluate  the  performance  of 
naive  classification  in  general  medicine. 


Actual 


Predicted 


Figure  BJ-31 
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In  order  to  test  the  naive  classifier's  performance  in 
general  medicine,  we  started  by  taking  a  subset  of  the  20 
main  CPT  categories.  Figure  BJ-31  shows  the  confusion 
matrix  for  the  initial  test.  We  took  8  of  the  main  groups 
with  65  or  more  instances,  and  used  35  instances  of  each 
group  to  build  the  training  set.  We  then  tested  30  other 
instances  for  each  separate  group. 

The  classifier,  which  works  by  converting  each  ICD9 
code  into  its  description  (e.g.  414.9  ==>  "Chronic  Ischemic 
Heart  Disease  Unspecified"),  yielded  very  promising 
results.  Note  that  we  used  only  exact  matches  of  words, 
which  meant  we  did  not  consider  words  with  a  common  root 
but  different  suffixes  to  be  the  same. 

The  next  step  was  to  observe  the  classifier's 
performance  when  using  more  than  8  of  the  groups,  and  more 
than  30  test  cases.  Since  the  classifier  seems  to  work 
best  with  an  evenly  distributed  data  set,  we  built  a  model 
where  each  category  had  200  instances  in  its  training-set, 
and  200  instances  in  its  testing-set.  Since  at  least  400 
items  were  required  for  each  category  in  this  test,  the  15 
categories  with  400  or  more  items  were  selected  for  use. 
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Figure  BJ-32 
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Figure  BJ-33 


Figures  BJ-22  and  BJ-23  show  remarkably  good  results. 
With  the  exception  of  the  Laproscopic,  Integumentary,  and 
Digestive  categories,  the  classifier  correctly  identified 
70%  to  90%  of  its  test  items.  Figure  BJ-33  shows  both  the 
actual  confusion  matrix  resulting  from  the  test  and  the 
percentage  in  each  actual  vs.  predicted  category. 

Again,  in  this  test,  only  exact  matches  of  words  were 
considered.  Words  with  common  roots  were  still  being 
recognized  separately.  Therefore,  it  seemed  possible  that 
the  three  challenging  categories  (Laproscopic, 
Integumentary,  and  Digestive)  might  work  out  better  if  the 
words  were  grouped  by  root,  as  opposed  to  all  being  treated 
as  distinct.  In  order  to  test  a  model  that  considered  some 
words  with  common  roots  to  be  equal,  a  classifier  was  built 
that  recognized  only  the  first  n  letters  of  every  word. 
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Figure  BJ-35 

Figures  BJ-34  and  BJ-35  show  the  results  of 
considering  all  words  beginning  with  the  same  four  letters 
to  be  the  same.  This  method,  which  attempted  to  sort  words 
by  root,  did  not  seem  to  make  much  difference  compared  to 
the  exact  match  method.  Other  tests  were  then  run  using 
the  first  5,  6,  and  7  letters  of  words,  but  those  made  even 
less  difference  than  the  initial  4-letter  root  test.  As 
seen  in  Figure  BJ-36,  this  root  method  inaccurately  grouped 
some  very  different  words  together: 
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(with  without) 

(procedure  proctocolitis) 

(hearing  heart) 

(polymorphonuclear  polyostotic  polyneuritis...) 

(postsurgical  postpartum  postmenopausal...) 

Figure  BJ-36 

Despite  these  obvious  mistakes,  the  results  didn't 
change  much.  The  Laproscopic  category  improved  slightly, 
and  the  overall  performance  was  improved  by  0.7%.  It 
seems,  in  order  to  effectively  group  words  by  root,  an 
electronic  thesaurus  may  be  required. 

Testing  Sub-Categories  of  the  Cardiovascular  Class 

The  next  task  was  to  classify  more  specific  CPT 
categories.  Under  the  each  of  the  main  CPT  categories, 
there  exist  more  specific  sub-categories.  We  took  the  22 
sub-categories  of  the  Cardiovascular  category  and  ran  tests 
to  see  whether  or  not  the  naive  classifier  could  handle 
more  specific  categorizations. 

The  first  test  used  every  Cardiovascular  sub-class 
with  at  least  20  related  consults: 
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Liga  58%  Ligation  and  Other  Procedures 

Sept  78%  Septal  Defect 

Hear  71%  Heart  /Lung  Transplantation  \ 

Expl  74%  Exploration 

Inte  0%  Intervascular  Cannulizat ion  or  Shunt 

Embo  41%  Embolectomy/ Thrombectomy 

Peri  68%  Pericardium 


Figure  BJ-37 

Figure  BJ-37  shows  that  the  initial  testing  of  more 
specific  categorization  had  some  success.  In  the  next 
step,  however,  we  wanted  to  try  using  an  unevenly 
distributed  training  set,  which  better  represented  the 
overall  distribution  of  the  sub-categories.  We  randomly 
assigned  half  of  the  data  to  the  training  set  and  the  rest 
to  the  test  set. 
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Figure  BJ-38 

This  test  (Figure  BJ-38)  actually  yielded  better 
results,  perhaps  due  to  the  more  realistic  distribution  of 
Cardiovascular  sub-categories. 

Next,  we  tested  using  a  more  evenly  balanced  training 
set  in  order  to  observe  the  difference  between  the  two 
methods  (even  vs.  uneven  data-set).  A  training  set  of  size 
50  was  used  for  each  sub-category  that  contained  50  or  more 
instances.  Eighteen  of  the  classes  could  be  tested  with 
this  method.  Figures  BJ-39  and  BJ-40  display  the  results 
of  this  test. 
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Figure  BJ-39 
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Under  60% 

(For  classes  with  at  least  20  test  cases) 


Figure  BJ-40 

This  method,  for  most  sub-categories,  was  somewhat 
more  effective.  However,  a  major  problem  with  this  method 
was  the  VENO  class.  VENO  includes  about  42%  of  all  the 
observations,  posing  a  problem  for  the  type  of  reasoning 
used  in  the  classifier. 
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Again,  we  tried  a  similar  test  method.  By  using  a 
data  set  of  size  25,  we  were  able  to  incorporate  all  22 


Cardiovascular  sub-categories. 


COMB 

BOON 

REPB 

REPA 

PERI 

OTHE 

SEPT 

AORT 

ARTE 

EMBO 

LIGA 

DIRE 

HEAR 

THOR 

THRO 

INTE 

BYPA 

EXPL 

CARD 

VASC 

VEH0 

PACE 

•  MML- 

0 

0 

0 

0 

0 

0 

TM 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

M  VC-I 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

l 

i '  i  :  h 

0 

1 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Q 

Zip 

0 

2 

I'L.  i 

0 

0 

0 

2 

0 

0 

0 

0 

0 

1 

0 

0 

0 

1 

0 

1 

0 

0 

□ 

0 

0 

0 

5 

PF“  T 

1 

0 

1 

0 

KB 

0 

0 

0 

2 

0 

0 

0 

0 

0 

D 

KB 

1 

1 

0 

l 

0 

30 

-  K 

KB 

0 

0 

0 

0 

BE! 

0 

□ 

0 

0 

0 

0 

2 

KB 

1 

D 

0 

2 

0 

0 

0 

B 

30 

:t:  I 

0 

0 

0 

0 

EX 

0 

0 

1 

1 

1 

0 

0 

2 

□ 

0 

0 

Q 

0 

0 

31 

T 

0 

0 

□ 

0 

0 

0 

KB 

mm 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

KB 

34 

!  h  l  J 

ES 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

1 

0 

55 

tr.r  . 

0 

4 

1 

0 

0 

0 

0 

0 

mm 

0 

1 

0 

0 

KB 

8 

7 

2 

0 

0 

0 

0 

68 

l 

0 

2 

3 

1 

0 

0 

0 

0 

4 

mm 

2 

1 

0 

2 

3 

3 

2 

0 

2 

0 

0 

77 

1-  -k 

5 

BEX 

1 

0 

1 

0 

0 

0 

0 

1 

l 

■B 

0 

2 

3 

1 

4 

0 

. 0~~ 

0 

0 

0 

95 

HLaI' 

KB 

0 

0 

0 

1 

KB 

0 

0 

0 

4 

0 

mm 

0 

0 

0 

0 

0 

KB 

0 

□ 

0 

105 

Ti-or 

KB 

BZI 

0 

0 

1 

0 

KB 

0 

0 

1 

1 

mm 

KB 

mm 

1 

0 

0 

0 

mm 

0 

2 

0 

132 

THF  ' 

KOI 

0 

0 

1 

0 

0 

0 

0 

0 

K9 

0 

0 

0 

0 

1 

mm 

0 

0 

□ 

2 

1 

133 

i  :;vt: 

0 

EB 

9 

0 

1 

0 

0 

0 

mm 

1 

0 

0 

0 

HJ 

KES 

i 

0 

0 

2 

1 

1 

154 

ev:;. 

3 

2 

4 

0 

0 

0 

0 

0 

1 

mm 

6 

7 

0 

l 

mm 

KB 

mi 

1 

0 

0 

KB 

0 

202 

6 

1 

7 

3 

11 

1 

0 

□ 

0 

29 

1 

3 

0 

0 

3 

0 

6 

m>v*u 

0 

0 

0 

0 

304 

30 

1 

1 

0 

4 

4 

2 

0 

0 

5 

0 

1 

4 

1 

2 

D 

0 

2 

507 

0 

2 

2 

568 

74 

9 

19 

11 

5 

13 

6 

0 

0 

49 

14 

12 

46 

0 

4 

182 

13 

13 

0 

333 

11 

5 

819 

2341 

4 

D 

0 

0 

1 

0 

0 

1 

5 

1 

0 

3 

0 

□ 

0 

1 

0 

1 

0 

45 

1 

2404 

65 

1 

33 

0 

1 

99 

1 

0 

1 

0 

0 

1 

42 

1 

6 

0 

23 

1 

4 

0 

3 

558 

840 

TOT 

2615 

66 

82 

30 

46 

133 

38 

32 

3 

190 

81 

102 

190 

72 

139 

301 

216 

247 

525 

337 

68 

576 

6089 

Figure  BJ-41 
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Figure  BJ-42 

The  results  shown  in  Figures  BJ-41  and  BJ-42  were  not 
as  good  as  those  shown  in  Figures  BJ-39  and  BJ-40.  While 
the  new  test  (Figures  BJ-41  and  BJ-42)  allowed  for  more 
categories,  it  did  not  have  as  big  of  a  data  set  as  the 
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test  in  Figures  BJ-39  and  BJ-40.  This  shows  the  struggle 
between  having  enough  information  in  the  training  set,  and 
being  able  to  assess  a  large  number  of  categories. 


Testing  Integumentary  Sub-categories 

After  evaluating  the  performance  of  the  naive 
classifier  with  specific  Cardiovascular  sub-categories,  we 
also  tried  similar  tests  for  the  Integumentary  class. 

In  these  tests,  we  randomly  split  each  sub-category 
into  two  separate  groups  -  groups  A  and  B.  These  divisions 
are  shown  in  Figure  BJ-43 . 
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Number 

Percent 

correctly 

correctly 
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classified 

Total 

classified 
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0 
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BIOPSY 

0 

2 
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B  URNS -LOCAL- TRE ATMENT 

0 

0 
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INTRODUCTION 
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10 

ion 

E 
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2 
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50h 
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OTHER-DESTRUCTICN-PROCEDURES 
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Oh 

G 

AD  J  AC  ENT- TISSUE- TRANS  F  E  R- OR- RE  ARRANGE  MENT 

0 

52 

Oh 
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REPAIR- INTERMEDIATE 
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16 

Sh 

I 

OTHER-FLAPS-AND-GRAFTS 

1 

18 

6h 

J 

OTHER-REPAIR-PROCEDURES 

11 

36 

3ih 

K 

EXC IS ICN-MAL IGNANT-LES IONS 

24 

41 

59% 

L 

EXC IS ION-BENIGN-LES IONS 

16 

61 

2  6% 

M 

FL APS-SKIN-AND-OR-DEEP-TISSUES 

2 

83 

2% 

N 

MISCELLANEOUS 

20 

22 

91% 

0 

NAILS 

0 

5 

0% 

P 

REPAIR-SIMPLE 

0 

6 

0% 

Q 

OTHER-BREAST-PROCEDURES 

1 

8 

13% 

R 

DESTRUCTION-BENIGN-OR-PREMAL IGNANT-LES IONS 

69 

81 

85% 

S 

INC IS ION- AND -DRAINAGE 

50 

117 

43% 

T 

PRESSURE-ULCERS-DECUB ITUS-ULCERS 

31 

38 

82% 

U 

REPAIR-COMPLEX 

9 

60 

3.5% 

V 

EXCISION 

167 

185 

90% 

U 

MOHS-HICROGRAPHIC-SURGERY 

4 

4 

100% 

X 

REPAIR- AND-OR-RECONSTRUCTION 

65 

119 

55% 

Y 

FREE-SKIN-GRAFTS 

1 

78 

1% 

Z 

EXCISICN-DEBRIDEHENT 

11 

122 

9% 

Total 

486 

1177 

41% 

Figure  BJ-45 

Both  tests  (Figures  BJ-44  and  BJ-45)  show  that  an 
unevenly  trained  data  set  performs  quite  poorly  for  the 
Integumentary  categories .  In  order  to  handle  this  more 
difficult  category,  we  used  the  even  approach.  By  using 
50-consult  training  sets  for  each  sub-category  with  at 
least  50  instances,  we  built  an  evenly  distributed  data 
set . 
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Figure  BJ-46 

Figure  BJ-46  shows  that  an  evenly  distributed  training 
set  works  better  for  the  difficult  Integumentary  category. 

The  exact  same  test  was  then  run,  this  time  on  the  Eye 
&  Adnexa  category.  Figure  BJ-47  shows  the  number  of 
instances  of  each  sub-category  in  the  Eye  &  Adnexa 
category.  Eleven  of  its  sub-categories  contained  50  or 
more  consults.  Therefore,  the  test  was  run  using  only 
those  particular  sub-categories. 


Eye  and  Adnexa  sub-class  counts 
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1 

Figure  BJ-47 
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127 
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387 
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80% 
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539 
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90% 

OVERALL 

13  95 
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76% 

Figure  BJ-48 

The  results  in  Figure  BJ-48  are  much  better  than  those 
in  the  Integumentary  class  (Figure  BJ-47) .  This  may  be 
greatly  due  to  the  fact  that  Eye  &  Adnexa  divisions  seem 
more  clear-cut  than  those  of  the  Integumentary  category. 


Naive  Tree  Method 

The  naive  classifier  can  only  produce  successful 
results  when  a  significant  amount  of  training  items  are 
available  for  each  category  in  the  range  of  categories  to 
be  classified.  The  classification  of  specific  CPTs, 
therefore,  is  quite  difficult,  because  the  range  of 
possible  CPTs  is  so  broad  that,  even  in  a  very  large  set  of 
data,  some  CPTs  may  be  involved  in  very  few,  if  any, 
instances.  As  a  result,  specific  CPT  classification  is  not 
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a  good  option  under  the  current  method  of  classification 
being  used. 

As  we've  already  seen,  the  naive  classifier  is  quite 
successful  in  sending  ICD9s  and  other  diagnoses  to  the  20 
main  CPT  categories.  Also,  we've  seen  that  given  an  ICD9 
that  belongs  to  C  (one  of  the  20  main  CPT  categories),  we 
can  often  successfully  classify  the  ICD9  into  the 
appropriate  sub-category  of  C.  Therefore,  an  effective 
solution  to  the  specific  CPT  problem  might  lie  in 
developing  the  method  by  which  the  classifier  handles  its 
levels  of  classes  rather  than  in  changing  the  naive  method 
of  the  classifier.  By  using  a  hierarchical  tree  method,  we 
can  utilize  the  naive  nature  of  the  classifier  at  different 
levels  of  classification  in  order  to  narrow  down  the  number 
of  categories  seen  at  each  level. 

The  structure  of  the  naive  tree  classifier  is  modeled 
after  the  hierarchical  structure  of  the  CPT  code  book. 
Under  each  of  the  20  main  CPT  categories,  another  two  mid¬ 
levels  of  categorization  exist;  and  the  bottom  level,  which 
consists  of  the  actual  specific  CPTs,  then  falls  under 
categories  in  one  of  each  of  the  other  three  levels. 

Denote  the  lowest  level  of  the  CPT  hierarchy,  which 
consists  of  specific  CPTs,  as  Level  3.  The  level  above 
that  will  be  labeled  Level  2,  and  the  one  above  it  Level  1. 
The  top  level,  which  consists  of  the  20  main  CPT 
categories,  will  be  referred  to  as  Level  0.  Figure  BJ-49 
shows  this  hierarchical  structure  of  the  CPT  codes. 
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Figure  BJ-49 


When  the  naive  tree  classifier  reads  in  an  item,  it 

first  classifies  the  item  at  the  most  general  level  (0)  . 
Let's  assume  that  out  of  the  20  main  categories  (Labeled  A- 
T) ,  the  classifier  first  selects  category  'A'  .  The  next 
step  in  the  process  would  be  to  again  classify  the  same 
item,  using  the  categories  in  Level  1.  However,  this  time, 
the  classifier  will  filter  out  all  categories  that  aren't 
descendants  of  the  Level  0  category,  'A' .  The  classifier 
will  then  proceed  down  the  tree  and  finally  select  from  a 
small  group  of  similar  CPTs . 

This  method  takes  advantage  of  the  fact  that  levels 

higher  in  the  tree  are  more  general,  and  are  easier  to 
classify.  It  also  allows  the  classifier  to  choose  from  a 
smaller  group  of  categories,  which,  as  we've  seen,  improves 
its  performance. 

Testing  was  done  on  the  hierarchically  structured  Eye 
and  Adnexa  category,  which  is  one  of  the  20  main  CPT 

categories.  For  testing  purposes,  we  assumed  that  the 

classifier  would  have  chosen  Eye  &  Adnexa  to  be  the  Level  0 
category.  We  then  tested  the  performance  for  each  level  of 
classification: 
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Actual 


Predicted 


Ocular 

Adnexa 

Eyeball 

Posterior 

Segment 

Anterior 

Segment 

TOTAL 

Ocular  Adnexa 

78 

13 

2 

4 

97 

Eyeball 

10 

93 

3 

5 

111 

Posterior  Segment 

12 

136 

628 

31 

807 

Anterior  Segment 

6 

65 

19 

732 

822 

TOTAL 

106 

307 

652 

772 

1837 

Figure  BJ-50 


Accuracy 


Ocular  Adnexa 

EYE  AND  ADNEXA  »  OCULAR  ADNEXA 

80% 

Eyeball 

EYE  AND  ADNEXA  »  EYEBALL 

84%  | 

Posterior  Segment 

EYE  AND  ADNEXA  »  POSTERIOR  SEGMENT 

78% 

EYE  AND  ADNEXA  »  ANTERIOR  SEGMENT 

89% 

Figure  BJ-51 


These  results,  at  Level  1  (Figures  BJ-50  and  BJ-51)  were 
very  successful.  The  next  step  in  the  complete  CPT 
classification  process  is  to  classify  the  same  text,  using 
only  the  sub-specialties  under  the  more  general  sub¬ 
specialty  that  was  just  selected.  The  classifier  then 
continues  to  move  down 
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(defun  menu-patients-on-click  (dialog) 
(declare  (ignore-if-unused  dialog)) 
(show-patients-dialog  (parent  dialog)) 
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(defun  menu-special ists-on-click  (dialog) 
(declare  (ignore-if-unused  dialog)) 
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(close  (e-board)) 

(close  (classify-dialog)) 
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(when  (or  (not  dialog) 

(not  (windowp  dialog))) 

(setq  dialog  (make-e-board  : parent  window)) 
(setf  (tdent-e-board-dialog  window)  dialog) 
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(declare  (ignore-if-unused  dialog  widget)) 
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(select-window  (classify-di alog)) 


(defun  string-to-symbol  (the-string) 

(if  (equal  ""  (string-trim  ' (#\Space)  the-string))  nil 
(with-input-f rom-stri ng  (s  the-string)  (read  s)))) 
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(defun  list-to-string  (the-list) 
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(if  (null  (edr  the-list))  (pri nc-to-stri ng  (car  the-list)) 

(concatenate  'string  (pri nc-to-stri ng  (car  the-list))  "  "  (list-to-string  (edr  the-list))))) 
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(defun  files-dialog-load-on-click  (dialog  widget) 
(declare  (ignore-if-unused  dialog  widget)) 

(let  ((file-list  nil)) 

(setf  file-list 

(find-widget  :file-list  dialog)) 


(defclass  patients-di alog  (dialog) 
((pati ent-i nf o-di al og 

: accessor  patient-info-dialog 
:initform  nil) 
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(defclass  consul t-info-2  (dialog) 
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(defun  consult-info-2-save-button-on-click  (dialog  widget) 
(declare  (ignore-if-unused  dialog  widget)) 
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(state  (state  location-instance)) 
(country  (country  location-instance)) 
(codel  (codel  location-instance)) 

(code2  (code2  location-instance)) 
(modality  (modality  location-instance))) 
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(defclass  locations-dialog  (dialog) 
((location-info-dialog 

:accessor  location-info-dialog 
rinitform  nil))) 
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(defclass  yellow-pages-dialog  (dialog) 
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(defun  cl assify-dialog-consul t-val ue-on-change-1  (widget  new-value  old-value) 
(declare  (ignore-if-unused  widget  new-value  old-value)) 

(cl ear-al 1 -but-consul t-val ue) 
t)  ;  Accept  the  new  value 
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(defstruct  dataset 

(number-of-i terns  0) 

(classes  NIL) 

(class-counts  (make-hash-table  :test  #'eql)) 

(words  NIL) 
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(/  (dataset-class-count  CLASS  DATASET)  (dataset-number-of-items  DATASET)) 


(defun  class-posterior-wi thout-word  (CLASS  WORD  DATASET) 

(setq  num-dont-use  (-  (dataset-class-count  CLASS  DATASET) 

(dataset-class-word-count  CLASS  WORD  DATASET))) 
(/  (+  num-dont-use  1)  (+  (dataset-class-count  CLASS  DATASET)  2)) 


<  r-  <u  o 
-J  p  cr  w  -a 

u  r-  a>  W 

v-/  3  w 


(defun  classify  (SENTENCE  WORDS  DATASET);  MIN-PM) 


maxClass 
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(defun  meaningless-word-p  (word) 
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(progn 

(push  (list  word  pet)  return-list) 
(push  word  wordz) 


(defun  sig-words-to-string  (class  text  data) 

(setq  return-string  NIL) 

(setq  sig-word-li st  (significant-words  class  text  data)) 

(dolist  (spec  sig-word-li st) 

(setq  return-string  (concatenate  'string  return-string  (list-to-string  spec) 
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(format  t  ”~A"  s-text) 

(read) 

(setq  minn  equ) 

(setq  min-text  s-text) 

(format  t  "change:  ~A~%"  minn) 


(setq  pred-class  (car  (car  (classify  s-text  (dataset-words  data)  data)))) 
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(format  t  "Prostho:  -A/~A~%Oral-Path:  ~A/~A~%"  prostho-c  prostho-t  oral-path-c  oral-path-t) 


